What techniques have been proposed to generate counterfactual explanations for NLP models?
In NLP, counterfactuals refer to alternative texts
that describe a different outcome than what is en-
coded in a given factual text. Prior works (Verma
et al., 2020) have focused on developing methods
for generating counterfactuals that adhere to several
key properties, including:
• Validity: the generated counterfactuals should
encode a different label from the original text.
• Closeness: the changes made to the text should
be small, not involving large-scale rewriting of
the input.
• Fluency: the generated counterfactuals should
be coherent and grammatically correct.
• Diversity: the method should generate a wide
range of counterfactuals with diverse character-
istics, rather than only a limited set of variations.
While many methods for automatic counterfac-
tual generation exist (Wu et al., 2021; Robeer et al.,
2021; Dixit et al., 2022), our work is mostly re-
lated to MiCE (Ross et al., 2021), which generates
counterfactuals in a two stage process that involves
masking the top-k tokens with the highest ℓ1 gra-
dient attribution of a pre-trained classifier, and in-
filling tokens for masked position with a T5-based
model (Raffel et al., 2020)
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Counterfactual data augmentation (CDA) -- i.e., adding minimally perturbed inputs during training -- helps reduce model reliance on spurious correlations and improves generalization to out-of-distribution (OOD) data. Prior work on generating counterfactuals only considered restricted classes of perturbations, limiting their effectiveness. We present COunterfactual Generation via Retrieval and Editing (CORE), a retrieval-augmented generation framework for creating diverse counterfactual perturbations for CDA. For each training example, CORE first performs a dense retrieval over a task-related unlabeled text corpus using a learned bi-encoder and extracts relevant counterfactual excerpts. CORE then incorporates these into prompts to a large language model with few-shot learning capabilities, for counterfactual editing. Conditioning language model edits on naturally occurring data results in diverse perturbations. Experiments on natural language inference and sentiment analysis benchmarks show that CORE counterfactuals are more effective at improving generalization to OOD data compared to other DA approaches. We also show that the CORE retrieval framework can be used to encourage diversity in manually authored perturbations
[{"first":"Tanay","last":"Dixit","middle":[]},{"first":"Bhargavi","last":"Paranjape","middle":[]},{"first":"Hannaneh","last":"Hajishirzi","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]}]
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences. We show that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three different tasks (with around 70% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts.
[{"first":"Tongshuang","last":"Wu","middle":[]},{"first":"Marco Tulio","last":"Ribeiro","middle":[]},{"first":"Jeffrey","last":"Heer","middle":[]},{"first":"Daniel","last":"Weld","middle":[]}]
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Humans have been shown to give contrastive explanations, which explain why an observed event happened rather than some other counterfactual event (the contrast case). Despite the influential role that contrastivity plays in how humans explain, this property is largely missing from current methods for explaining NLP models. We present Minimal Contrastive Editing (MiCE), a method for producing contrastive explanations of model predictions in the form of edits to inputs that change model outputs to the contrast case. Our experiments across three tasks--binary sentiment classification, topic classification, and multiple-choice question answering--show that MiCE is able to produce edits that are not only contrastive, but also minimal and fluent, consistent with human contrastive edits. We demonstrate how MiCE edits can be used for two use cases in NLP system development--debugging incorrect model outputs and uncovering dataset artifacts--and thereby illustrate that producing contrastive explanations is a promising research direction for model interpretability.
[{"first":"Alexis","last":"Ross","middle":[]},{"first":"Ana","last":"Marasovi'c","middle":[]},{"first":"Matthew","last":"Peters","middle":["E."]}]
Generating Realistic Natural Language Counterfactuals
Counterfactuals are a valuable means for understanding decisions made by ML systems. However, the counterfactuals generated by the methods currently available for natural language text are either unrealistic or introduce imperceptible changes. We propose CounterfactualGAN: a method that combines a conditional GAN and the embeddings of a pretrained BERT encoder to model-agnostically generate realistic natural language text counterfactuals for explaining regression and classification tasks. Experimental results show that our method produces perceptibly distinguishable counterfactuals, while outperforming four baseline methods on fidelity and human judgments of naturalness, across multiple datasets and multiple predictive models.
[{"first":"Marcel","last":"Robeer","middle":[]},{"first":"Floris","last":"Bex","middle":[]}]
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Contrast sets (Gardner et al., 2020) and counterfactual data (Kaushik et al., 2020) provide minimal input perturbations that change model predictions, and serve as an effective means to evaluate brittleness to out-of-distribution data (Wang et al., 2021). Counterfactual data augmentation (CDA) has shown to improve model robustness to OOD data and input perturbations (Geva et al., 2021;Wu et al., 2021;Paranjape et al., 2022;Khashabi et al., 2020). Alternate methods like debiasing data (Wu 1 Code at https://github.com/tanay2001/CORE Control activities happen everywhere in the entity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
There is a lack of control activities everywhere in the entity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Control activities exist everywhere in the company.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Control activities occur at all levels and functions of the entity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Control activities happen nowhere in the entity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Control activities do not happen everywhere in the entity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Internal activities happen at some levels in the entity Figure 1: Diverse counterfactuals are generated for an MNLI example. The red arrow represents the most trivial way of generating a counterfactual hypothesis, while the violet arrows depict several other perturbations that intervene on different predictive features. et al., 2022) have also shown promising results on improving model robustness, but in this work we focus on CDA strategies. Recently, Joshi and He (2022) find that diversity in the set of perturbations of different predictive features is key to the effectiveness of CDA (see Figure 1). In this paper, we introduce COunterfactual Generation via Retrieval and Editing (CORE) -retrieval augmented generation framework for creating diverse counterfactual perturbations. CORE combines dense retrieval with the few-shot learning capabilities of large language models, while using minimal supervision about perturbation type. Retrieval-augmented models (Guu et al., 2020;Lewis et al., 2020) learn to search over a dense index of a text corpus to condition generation on retrieved texts and are especially effective at improving the diversity of generated text for paraphrase generation (Chen et al., 2019) and style-transfer (Xiao et al., 2021). CORE uses this insight by learning to retrieve counterfactual excerpts from a large text corpus. Arbitrarily conditioning on these retrieved text excerpts to generate a rich set of counterfactual perturbations, without explicit supervision, can be challenging (Qin et al., 2022). Instead, CORE uses few-shot prompting of massive pretrained language models, which is found to be effective at controlled generation tasks like arbitrary style-transfer (Reif et al., 2022). CORE prompts GPT-3 (Brown et al., 2020;Wei et al., 2022) with a few demonstrations of using these excerpts for counterfactual editing.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
The CORE retriever is a transformer-based biencoder model trained using contrastive learning (Le-Khac et al., 2020) on a small set of humanauthored counterfactuals for the task. For each training example, CORE retrieves excerpts from an unlabeled task-related corpus that bear a labelflipping counterfactual relationship with the original input instance. Retrieval may extract excerpts that have significant semantic drift from input text, while still containing relevant counterfactual phrases (Table 1). Using prompts, the CORE GPT-3 editor generates counterfactual edits to the input conditioned on the retrieved excerpts (and the original inputs). The prompts consist of instructions and a few demonstrations of using the retrieved text for editing. Unlike prior work that use rule-based (Ribeiro et al., 2020) or semantic frameworks (Wu et al., 2021;Ross et al., 2022) and restrict perturbation types, CORE uses naturally occurring data to encourage perturbation diversity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Intrinsic evaluation of CORE counterfactuals demonstrates a rich set of perturbation types which existing methods like Wu et al. (2021) generate (Table 7) and new perturbation types (Table 5) with more diverse outputs (Table 6), without explicit supervision. Our extensive data augmentation experiments and analyses show that the combination of retrieval and few-shot editing generates data for CAD that is effective in reducing model biases and improves performance on out of distribution (OOD) and challenge test sets. Perturbing only 3% and 7% of the data for NLI and Sentiment analysis respectively, we achieve improvements up to 4.5% and 6.2% over standard DA (Tables 2,3). Additionally, we show that CORE's learned retriever can assist humans in generating more diverse counterfactuals, spurring their creativity and reducing priming effects (Gardner et al., 2021).
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Our counterfactual retriever is based on the dense passage retrieval (DPR) framework (Karpukhin et al., 2020). CF-DPR retrieves similar instances from a large unlabeled corpus that have different labels. Formally, given a training set, N (x) = {(x 1 , y 1 ), (x 2 , y 2 ), . . . , (x n , y n )}, for a text classification task and a large corpus S, CF-DPR retrieves samples C(x) = {x 1 ,x 2 , . . . ,x n } from S such that the associated labels for samples in C(x) are not the same as the corresponding labels in N (x). Specifically,ŷ i = y i ∀i ∈ (0, n), whereŷ i is the class label for retrieved samplex i . In Figure 2, for the input "The most opaque , self-indulgent and just plain boring sequel to End Game.", CF-DPR retrieves the excerpts "The most delightful trailer ever" and "Spot on awesomeness in this". Training We use the same contrastive learning objective as Karpukhin et al. (2020) to train the bi-encoder model. It consists of two independent BERT (Devlin et al., 2018) encoders: a query encoder P that encodes x i in N (x) as p i and a document encoder Q that encodes text excerpts in S as q i . To train the bi-encoder, we use a small seed training dataset
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
of size m contain-ing m < |N (x)| positive and negative retrieval samples. For a given training instance q i , we use its corresponding positive sample p + i and hard negatives p − i 's to optimize the following loss function.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
(1)
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
To model the task of counterfactual retrieval, for each training instance x i = q i , we use the corresponding counterfactual instance as the positive sample (p + i ) and use paraphrases of q i as the hard negative (p − i ). Positive samples can be obtained from a seed dataset consisting of manually authored counterfactuals for existing NLU datasets like IMDb, and SNLI (Kaushik et al., 2020;Gardner et al., 2020). This manual data is of the form T = {(q 1 , p + 1 ), (q 2 , p + 2 ), . . . , (q m , p + m )}. We make use of the diverse paraphraser model (Krishna et al., 2020) that generates paraphrases as hard negatives for {q 1 , q 2 , . . . , q m }, {p − 1 , p − 2 , . . . , p − m }. Contrastive training pulls counterfactual samples p + i closer to q i and pushes semantically identical sentences p − i away from q i . We show that this counterfactual retrieval framework can be used to retrieve counterfactuals for tasks with only a small amount of seed training data ( §4.1). Additional details about training and evaluation of the trained CF-DPR are in Appendix A.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Inference We create S for a specific task dataset using (1) text corpora that have similar domains as that dataset and (2) other datasets for the same task. For instance, for sentiment analysis over IMDb, we use a large (1.6 million) corpus of movie reviews from Amazon (McAuley and Leskovec, 2013) and the Yelp review dataset (Asghar, 2016). We encode the entire search corpus using the trained document encoder and index it using FAISS (Johnson et al., 2019). For every input training instance x i , we retrieve top k relevant counterfactuals
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We refer to these as CF-DPR counterfactuals.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
The retrieved counterfactuals often contain relevant phrases that perturb predictive features in different ways ("opaque" → "delighful", "boring" → "spoton awesome" in Figure 2), but are typically not a minimally edited version of the training sample. "The most delightful trailer ever" has the opposite sentiment as the original review, but is about another entity. To incorporate perturbation diversity while controlling for minimality, CORE uses an editor module. The Editor takes the training sample and retrieved counterfactuals as input and generates a minimally edited counterfactual. This involves selecting parts of the retrieved text that maybe useful in flipping the original text's label and then seamlessly integrating them into the original input. This fine-grained instance-specific style-transfer task can be hard to find supervision for.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We use GPT-3 (Brown et al., 2020) since it has been successfully used for fine-grained style transfer with few-shot supervision in prior work Reif et al. (2021). For instance, GPT-3 can learn from natural language constraints in its prompt, such as "include the word balloon," for constrained decoding. To prompt GPT-3, we make use of the set of instructions that were provided in Kaushik et al. (2020) to instruct crowd workers to author counterfactuals. We also append four human authored demonstrations of incorporating retrieved data, depicting various perturbation types.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Following Reif et al. (2022), we simplify our demonstrations by extracting keywords from the retrieved samples and providing them as token-level constraints in the prompt. To encourage the model to perturb certain classes of words, we remove determiners, conjunctions, and punctuation from the retrieved samples and tokenize the rest of the input into a list of keywords: [w 1 , . . . , w n ]. The resultant demonstration in our prompt thus becomes: "Input: ... Words to use: [w 1 , . . . , w n ], Edited:...". This is motivated by Wu et al. (2021)'s observation that perturbing certain classes of words (like preposition and adjectives) leads to better counterfactual generation. More details about the prompt construction are in Appendix A.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We generate CORE counterfactuals for two taskssentiment classification of movie reviews and natural language inference. We describe task-specific details about CORE training and inference below.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Task Dataset (N (x)) We create CORE counterfactuals for the IMDb movie review dataset (Maas et al., 2011), which has been used to manually create contrastive data (Kaushik et al., 2020;Gardner et al., 2020). This dataset presents unique challenges due to the longer average length of reviews (233 words), that existing counterfactual generation techniques (Wu et al., 2021)  Our initial experiments indicated that indexing full movie reviews did not yield good CF-DPR performance, owing to more dense retrieval noise when encoding longer contexts (Luan et al., 2021). Hence, we sentence tokenize the reviews and index each sentence independently. The search corpus contains approximately 8 million sentences. Subset to augment N (x) We generate CORE counterfactuals for the same subset of 1.7k reviews from IMDb chosen by Kaushik et al. (2020) in order to make fair comparisons in §5. Key-word list [w 1 , . . . , w n ] We observed that several sentences in the review contain descriptive information (plot, cast, etc) that don't convey sentiment. In Kaushik et al. (2020) too, humans edited a few sentences to change the sentiment of a long review. We extract these sentences containing polarity features, use them as queries to CF-DPR, and retrieve their counterfactuals. This selection strategy can also be extended to any review by learning a sentence selection model on annotated data (DeYoung et al., 2020). The keyword list is extracted from retrieved outputs as described in §3.2. Note IMDb Review: It's the worst movie I've ever seen. The action is so unclear, work of cameras is so poor, actors are so affected ... and this lamentable 5 minutes of Arnie on the screen. My advice from the bottom of my heart -don't watch it unless you like such a low class torture. Sentiment: Negative
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
GPT-3 CORE Probably the dozenth time I've seen it. This is another such effort. This is one spectacular production. that is definitely worth taking advantage of.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
It's one of the best movies I've ever seen. The action is so clear, work of cameras is so great, actors are so unaffected ... and this wonderful 5 minutes of Arnie on the screen. My advice from the bottom of my heart -watch it unless you don't like such a high class torture.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
It's the best movie I've ever seen. The action is so clear, work of cameras is so great, actors are so professional ... and this spectacular 5 minutes of Arnie on the screen. My advice from the bottom of my heartdefinitely watch it if you like such a high class production.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Premise: and my my uh taxes are a hundred and thirty five. Hypothesis: My taxes are $135 (Entails) CF-DPR GPT-3 CORE My sister spent over $2, 000 on a computer that she'll probably never use.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
My taxes are not $135.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
My taxes are probably over $135
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Premise:After the battle against the slave lord, the group needed it. Hypothesis:The slaves battled the group. (Contradicts) CF-DPR GPT-3 CORE The slaves never fought against their tyrannical captors.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
The slaves battled the slave lord. The slaves fought against their captors. that we do not impose any restrictions on the editor regarding which sentences to edit.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Task Dataset N (x) We focus on MNLI (Williams et al., 2018), a popular NLI dataset that tests for complex language reasoning. CF-DPR training data p + i , p − i We use the inherent paired nature of MNLI. In MNLI, given a premise, annotators are asked to manually write one sentences that entail, contradict or are neutral to the premise. These three hypotheses serve as mutual counterfactuals. In this work, we limit counterfactual perturbations to entailment(E)→contradiction(C) and vice-versa, to simplify the different permutations of positives and negatives required for CF-DPR training. We find that including the neutral class leads to increasingly noisy retrieved data, as the semantic differences between neutral class and the other two NLI classes are subtle and hard to distinguish. In Equation 1, q i is generated by concatenating the premise and hypothesis separated by the special token [SEP]. For every such input, p + i is a hypothesis from the counterfactual class, while p − i are diverse paraphrases of the original hypothesis. Task-specific corpus (S) is constructed by combining the following NLI datasets (Williams et al., 2018;Bowman, Samuel R. and Angeli, Gabor and Potts, Christopher, and Manning, Christopher D., 2015;Liu et al., 2022) in addition to the source corpus 2 that was used to generate premises in MNLI. We also include tokenized wikipedia articles (Merity et al., 2016) as several domains in MNLI (Eg. travel, government) are related. The search corpus contains approximately 7 million text excerpts. Subset to augment N (x) To compare with the state-of-the-art data augmentation technique for MNLI, WaNLI (Liu et al., 2022), we choose a subset of the MNLI dataset for augmentation based on their selection strategy. WaNLI uses dataset cartography (Swayamdipta et al., 2020) to select the most ambiguous examples -where model confidence across training epochs is low and variance is high -for augmentation. We generate 9.5K additional examples in two classes. Cross-Encoder We incorporate a re-ranker module to boost retrieval results for MNLI, that uses a cross-encoder architecture (Thakur et al., 2021) to jointly encode query q i and top-K documents retrieved by the bi-encoder. Given a q i and K retrieved sentences from the bi-encoder, the re-ranker learns to classify them as positive or negative. During inference, bi-encoder outputs are re-ranked based on their cross-encoder probability. The cross encoder is trained on the binary classification task on the same seed dataset as the bi-encoder. Retrieval performances are reported in Appendix C.   
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We first list the various augmentation strategies against which we compare CORE ( §5.1.1) followed by data augmentation results ( §5.1.2). In ( §5.2) we highlight the need for a counterfactual retriever, and in ( §5.3) we intrinsically evaluate CORE counterfactuals on their quality and perturbation diversity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We augment the full training datasets with |N (x)| CORE counterfactuals. We fine-tune DeBERTa base model (He et al., 2021) on the combined dataset.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We compare augmentation with CORE to strong DA baselines and ablations to different parts of our data generation pipeline. For all the strategies we augment with the same number of instances.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
In order to evaluate the effectiveness of CDA over adding more indomain data, we add the same number of in-domain training examples. For sentiment classification, we add 1.7k movie reviews randomly sampled from the Amazon Movie reviews dataset (McAuley and Leskovec, 2013). For MNLI, we use the WaNLI (Liu et al., 2022) dataset constructed using MNLI, that showed impressive OOD gains. We randomly sample 9K reviews from E and C classes in WaNLI.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Augmentation For MNLI, we compare against Tailor (Ross et al., 2022) using the SWAP_CORE control code as it results in label flipping perturbations. We do not compare with Polyjuice (Wu et al., 2021) as it requires complete human relabelling hence it would not be a fair comparison. Human generated data For IMDb, we also compare against human-authored counterfactual data, CAD (Kaushik et al., 2020). Amazon reviews, WaNLI, and CAD involve human supervision for dataset construction and are generally noise-free, unlike CORE data (see §5.3 for noise estimates). GPT-3 Counterfactuals To ablate the effect of conditioning on retrieval, the GPT-3 counterfactual baseline edits inputs into counterfactuals based only on a prompt consisting of task instruc-tion and demonstrations with NO keyword lists. CF-DPR Counterfactuals To ablate the GPT-3 few-shot editor in the CORE pipeline, the CF-DPR counterfactual baseline that uses retrieved outputs of CF-DPR as counterfactuals for augmentation and does no few-shot editing. See Appendix A for more task-specific details of this baseline.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
IMDb Table 2 shows that using just 1.7k (7%) human-annotated contrastive data to train the CF-DPR model, followed by the GPT-3 editing step, CORE obtains a performance gain of up to 6.2% over the IMDb un-augmented baseline. We compare against human-authored Kaushik et al. (2020) (CAD) clean examples in the CAD dataset and MiCE, an automatic counterfactual editing technique. We find that CORE is especially effective at OOD improvements on the Senti140, SST2 and Yelp datasets (by 3.66% points on average); despite augmenting data with noisy labels and with no explicit human supervision for editing. We hypothesize that this may be because of priming biases in human-authored counterfactual data (Bartolo et al., 2021b) and more diversity in CORE counterfactuals. A more detailed analysis is presented in §5.3.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Standard in-domain augmentation (i.e. bearing no counterfactual relationship with original data) is not as effective at improving performance on OOD and contrastive sets ( Table 2,Table 3), thus highlighting the importance of counterfactualy generated data.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Ablating individual parts of the pipeline, i.e GPT-3 based editing and CF-DPR retrieval -we observe that individual components are less effective than the combination of both techniques in CORE. CF-DPR retrieval may create reviews that incoherent and have a large semantic shift from the original review. Though the independent GPT-3 editor generates reviews that are minimally edited, they may contain recurring perturbation types (Table 1), limiting its efficacy for CDA. Combining the two helps overcome individual drawbacks.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
MNLI Using the DeBERTa-base model, with just augmenting 9.5k (3%) of the data we get improvements of up to 4% over the unaugmented dataset (Table 3). The improvements are particularly on the WaNLI evaluation set and adversarially designed test sets HANS and ANLI. Compared to Tailor, CORE achieves a 2% and 1.3% gain on HANS and SNLI, respectively. Once again, re-   trieval and GPT-3 editing are not as effective individually as the combination of the two in CORE. Independent GPT-3 generated counterfactuals are biased towards simple perturbations ( §5.3) and augmenting with these counterfactuals hurts performance on HANS and on Lexical Inference test set (LI_LI). CORE is also competitive with WaNLI data augmentation, which is noise-free in-domain data constructed with the same subset selection strategy as used by CORE. We also consider augmenting with both CORE and WaNLI counterfactuals, which results in orthogonal benefits and improvements on SNLI and ANLI datasets.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
To understand the importance of a trained counterfactual retriever, we ablate the retrieval stage of CORE by using a simple TF-IDF based retriever. We use the same GPT-3 Editor to generate the counterfactuals, only replace the CFDPR counterfactual retriever with a TF-IDF retriever. The TF-IDF based CORE counterfactuals exacerbate biases as seen in Table 4, significantly hurting performance by 8% and 3% on HANS and Adv. NLI, full table in Appendix (Table 15). This highlights the need 2. Seldom does a contrived screenplay comes across that is ever so observant of its characters.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
1. As a film director , LaBute continues to improve . 1.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
[lex] As director, LaBute continues to crap.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
1. As a film director, LaBute is not as good as one might hope. 2. The movie is a mess from start to finish . 2.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
[lex] It is a wonderful movie from start to finish. 2. I went to the movie weeks ago and enjoyed it from start to finish. Table 5: Examples of perturbations on SST2 data. Polyjuice often uses restricted patterns like perturbing "not" or using antonyms, unlike CORE's non-trivial perturbations conditioned on retrieved data. for a counterfactual retrieval module that retrieves counterfactual words/phrases in an unconstrained manner without limiting potential semantic edits.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
To analyze the source of its empirical gains of CORE in §5.1.2, we evaluate the label correctness, closeness, and perturbation diversity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Label Correctness Unlike Polyjuice (Wu et al., 2021) CORE encourages label flipping behavior during the generation process, possibly at the cost of label correctness. We quantify label correctness of the generated data by manually annotating a sample of 100 data points for IMDb and MNLI. Our analysis show that noise levels are 41% for IMDb and 40% for MNLI. CORE's augmentation benefits ( §5.1.2) persist even when compared to noise-free data (CAD for IMDb and WaNLI for MNLI), underscoring the importance of diversity.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
The intuition behind CORE is that effective CDA requires perturbation of various kinds so that perturbation bias is not exacerbated (Joshi and He, 2022). To measure this effect, on the IMDb dataset, we compare with CAD and another prior work on CAD generation, MiCE (Ross et al., 2021). In Figure 3, for the augmented subset we plot the probability of a token predicting class labels as a function of token count. We observe that CAD (Kaushik et al., 2020) has several outliers tokens that have strong bias towards a label, compared to CORE.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Since counterfactuals are meant to be minimally edited instances of the original input (Gardner et al., 2020), we analyse the closeness of the generated counterfactuals to the original text. To check how close these generated counterfactuals are to the original instance, we measure the Levenshtein edit distance between the two. To quantify diversity, we also measure self-BLEU (Zhu et al., 2018). The self-BLEU score is computed between edited counterfactual and the original text on the IMDb dataset. Since self-BLEU and Levenshtein are opposite in nature, we ideally want the counterfactuals to strike a balance between the two (Wu et al., 2021). As shown in in Table 6, CORE counterfactual are more diverse (lower Self-BLEU) which is at a small cost of edit distance when compared to CAD and MiCE.   MNLI  CORE  691  1303  1320  1073  138  92  79  4577  GPT-3  1400  705  1169  2184  290  95  94  3545   SST2  CORE  81  140  202  174  49  5  60  1305  Polyjuice  161  91  209  362  26  17  66  1076   Table 7: List of perturbation types detected using the set of heuristics from Wu et al. (2021). We can see that for both the task Sentiment and NLI, CORE covers all perturbation types without any explicit control code.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Supporting Human Annotation Prior work proposes aiding crowd-workers in the task of dataset creation using generative assistants (Bartolo et al., 2021a) or randomly sampled words (Gardner et al., 2021) to encourage creativity, leading to better quality and diversity in human-authored data. We analyze the impact of CF-DPR counterfactuals in encouraging humans to make diverse counterfactual perturbations to text. We design a controlled crowdsourcing experiment where 200 original MNLI examples from the validation set are shown to humans with and without the retrieved counterfactual sentences to aid them. Humanauthored counterfactuals conditioned on retrieved outputs display more diversity, with lower self-BLEU and higher Levenshtein distance compared to the control condition (Table 6). Qualitative differences between human authored counterfactuals in both conditions and details of the crowdsourcing are in the Appendix A.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We present CORE, a retrieval-augmented generation framework for creating diverse counterfactual perturbations for CDA. CORE first learns to retrieve relevant text excerpts and then uses GPT-3 few-shot editing conditioned on retrieved text to make counterfactual edits. CORE encourages diversity with the use of additional knowledge for this task, explicitly via retrieval and implicitly (parametric knowledge), via GPT-3 editing. Conditioning language model edits on naturally occurring data results in diversity ( §5.3). CORE counterfactuals are more effective at improving generalization to OOD data compared to other approaches, on natural language inference and sentiment analysis( §5.1.2). CORE's retrieval framework can also be used to reduce priming effects and encourage diversity in manually authored perturbations ( §5.3).
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
While CORE involves no human intervention, it is not completely accurate at performing label flipping perturbations and human re-labelling can be beneficial. Our framework uses a learned retriever which can be challenging to train when there are finer semantic differences between classes (e.g. neutral class) that need to be captured. Work on counterfactual generation has focused exclusively on English language text, and it would be an interesting future work to expand such frameworks for other languages.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Akari Asai and Hannaneh Hajishirzi. 2020. Logicguided data augmentation and regularization for consistent question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5642-5650, Online. Association for Computational Linguistics. A Implimentation details for CORE CF-DPR We make use of the open source implementation 3 of DPR to train the CF-DPR model. We train the CF-DPR model using 2 hard negatives, one the paraphrase and the second the query instance (hypothesis in case of MNLI and entire review for IMDB). All training was performed using 3 TITAN X (Pascal) GPUs. The average run time for CF-DPR for the MNLI task was 5hrs and for IMDB it was 1hr. Total emissions are estimated to be 4.19 kgCO 2 eq of which 0% was directly offset. Estimations were conducted using the Machine-Learning Impact calculator presented in (Lacoste et al., 2019). The training parameters used to train CF-DPR for the MNLI task are in Table 8 and for IMDB Table 9.   We experiment with several hyperparameters and find the quality of the hard negatives used to train the model played the most critical role in training. Adding more hard negatives didn't really help boost performance. Changing learning rate and warmup steps did not drastically affect the training process either. For evaluation we use the validation sets from (Kaushik et al., 2020) and MNLI validation-matched set for sentiment and NLI CF-DPR models respectively and reformat it as described in §4.1 and §4.2. For every instance in the validation set, we generate 30 random negatives and 30 hard negatives. Since we cannot generate 30 different paraphrases we make use of a semantic sentence retriever model (Gao et al., 2021) to retrieve semantically similar sentences from a text corpus. Finally following the DPR codebase we measure the top 1 accuracy on the validation set. For MNLI our CF-DPR model achieves an top 1 accuracy of about 70% while for IMDb around 45%.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
The cross encoder model was trained using the sentence-transformers 4 library. Each instance in the training dataset for MNLI contains a query instance which is a concatenated version of the premise and hypothesis separated by the [SEP] token, and two hypothesis with labels 1 and 0 respectively. We train the bert-base-model using the BCE loss. We train the model for 10 epochs with a batch size of 64 and learning rate 2e − 5. We evaluate every 2000 setps and save the model with the best evaluation loss. The cross encoder gave a validation accuracy of 91%.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
For Inference we encode the entire search corpus with the context encoder and index it using faiss. We approximate maximum inner-product search (MIPS) with an Inverted File Index (IVF) (Sivic and Zisserman, 2003) for faster retrieval. We use the IVFFlat index 5 as it helps improve the retrieval speed at a small cost of accuracy. We set the number of centroids (K) as 300 and n_probes to 30.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
GPT-3 few shot prompting We use the text-davinci-002 model as the Editor. The prompts to GPT-3 for IMDB is in Table 11 and for MNLI Table 10. For both the model settings we set the temperature parameter to 0.7 and Top p parameter to 1.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
DeBERTa finetuning All training was performed using 3 TITAN X (Pascal) GPUs. We evaluate every 400 steps and save the model with the best evaluation loss. We make use of the huggingface trainer APIs 6 for fine-tunning the models. It takes around 2 hrs to fine-tune the DeBERTa base model for MNLI and less than an hour for IMDb. The fine-tuned models were evaluated on publicly available validation/test sets.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
For MNLI in-domain performance we only report results on the mis-matched validation set as we observe that both the sets matched and mismatched had similar scores. For QNLI, SNLI we use the datasets that are part of the GLUE benchmark. For WaNLI, HANS and Diagonistics we use the same set of validation sets used in Liu et al. (2022) and for ANLI and LI_LI we use the official datasets provided by the authors. Since HANS and QNLI are binary NLI tasks (entailment and non-entailment), for measuring accuracy we consider both Neutral and Contradiction predictions as non-entailment.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
For IMDb we use all the official datasets available. For CAD we combine the validation and tests in-order to get more statistically significant results.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Crowd-sourcing Study In Section 5.3, we use CF-DPR outputs to aid crowd-sourcing of counterfactual edits. On the MNLI development set, we randomly sample 200 instances. We create two user interfaces for crowdworkers on Amazon Mechanical Turk to collect data under two conditions. In one condition, workers are shown the original instance (premise and hypothesis) and top-three retrieved counterfactuals provided by CF-DPR. They are asked to edit the hypothesis following brief instructions instructions and examples, shown in Figure 6. In the second case, they are just shown the original instance, no retrieved outputs. When revising examples, we asked workers to preserve the intended meaning through minimal revisions. Each instance is modified only once and different annotators are shown instances from both sets. Annotators were required to have a HIT approval rate of 90%, a total of 1,000 approved HITs. For the case where annotators were shown retrieved sentences, we found that annotator quality was quite poor, since annotators were not filtered by a qualification test to do the task. More generally, complex annotator tasks often require substantial training of crowd-workers (Bartolo et al., 2021a), which is outside the scope of this work. Instead, we recruit computer science graduate students (outside of the study) to get annotations for this task.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Two sentences are given, sentence 1 and sentence 2. Given that Sentence 1 is True, Sentence 2 (by implication), must either be (a) definitely True, (b) definitely False. You are presented with an initial Sentence 1 and Sentence 2 and the correct initial relationship label (definitely True or definitely False). Edit Sentence 2 making a small number of changes such that the following 3 conditions are always satisfied: (a) The target label must accurately describe the truthfulness of the modified Sentence 2 given the original Sentence 1. (b) In order to edit sentence 2, must only make use of one or two relevant words present in the provided list of words. (c) Do not rewrite sentence 2 completely, only a small number of changes need to be made. Here are some examples: original sentence 1: Oh, you do want a lot of that stuff. original sentence 2: I see, you want to ignore all of that stuff. initial relationship label: definitely False target label: definitely True List of words: ['correct', 'before', 'happen', 'already', 'more', 'saw', 'on'] modified sentence 2: I see, you want more of that stuff. original sentence 1: Region wide efforts are also underway. original sentence 2: Regional efforts have not stopped. initial relationship label: definitely True target label: definitely False List of words: ['talking', 'hold', 'put', 'caller', 'on'] modified sentence 2: Regional efforts are on hold at the moment. original sentence 1: yeah you could stand in there if you really wanted to i guess. original sentence 2: If you want you can sit there I guess. initial relationship label: definitely False target label: definitely True List of words: ['there', 'without', 'ants', 'stand', 'even'] modified sentence 2: If you want you can stand there I guess. original sentence 1: and uh every every opportunity there is to make a dollar he seems to be exploiting that. original sentence 2: He works to make money. initial relationship label: definitely True target label: definitely False List of words: ['work', 'pass', 'dollars', 'up', 'owe'] modified sentence 2: He passes up on opportunities to make money. Given a movie review and its sentiment. Edit the review making a small number of changes such that, the following two conditions are always satisfied: (a) the target label accurately describes the sentiment of the edited review (b) make use of only a few key words provided in the list of words to edit the review Do not remove or add any extra information, only make changes to change the sentiment of the review. Review: Long, boring, blasphemous. Never have I been so glad to see ending credits roll. Label: Negative List of relevant words: ['clean' , 'now', 'brought', 'perfect' , 'many' ,'interesting' ,'memories', 'sad'] Target Label: Positive Edited Review: Perfect, clean,interesting. Never have I been so sad to see ending credits roll. Review: I don't know why I like this movie so well, but I never get tired of watching it. Label: Positive List of relevant words: ['hate' , 'now', 'supposedly' , 'many' , 'memories', 'watching'] Target Label: Negative Edited Review: I don't know why I hate this movie so much, now I am tired of watching it. 
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Model microsoft/deberta-base learning rate 2e −5 number of epochs 3 per device batch size 32 max seq length 128 
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Model microsoft/deberta-base learning rate 2e −5 number of epochs 5 per device batch size 32 max seq length 128 
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
We show examples of the counterfactual editing of MNLI examples done by crowd-workers who were asked to independently edit instances and experts (CS graduate students) who were shown CF-DPRretrieved counterfactual instances in Table 17.  In addition to the examples present in Table 1 we also provide some more examples for MNLI in Table 18 and IMDb in Table 19.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
C Additional Analysis Z statistics scores Figure 3 depicts the competancy style plot for the IMDb augmented subset. In addition to that we also show the individual z scores in Table 14 SST-2 Results We also study the ability of CORE to generate counterfactuals on SST2 without being explicitly trained on SST2 data. We use the CF-DPR model trained on IMDb. The results and comparison with Polyjuice can be found in Table 16.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
CDA baselines For all the baselines we make use of the official implementations. For Tailor on MNLI, we use both no context and in-context swap-core perturbations.    Table 15: Accuracies of data augmentation for CORE and retrival ablation. We observe that using a simpler TF-IDF based retriver doesn't help improve performance across any task, in contrast to using a counterfactual retriever.  Premise: This northern beach of magnificent tan sand is most agreeably reached by boat. Hypothesis:The beach has beautiful sand. (Entailment) New Hypothesis: The sand of the beach is ordinary
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
New Hypothesis: The beach is impossible to reach by boat and the volcano has devastated the island.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Premise: You never call. Hypothesis: You rarely call on the phone, nor webcam. (Entailment) New Hypothesis: You always call or webcam New Hypothesis: You run up a big bill on your phone calling me! Premise:have that well and it doesn't seem like very many people uh are really i mean there's a lot of people that are on death row but there's not very many people that actually um do get killed Hypothesis: There are a lot of people on death row, but not that many actually are executed. (Contradiction) New Hypothesis: There are not many people on death row, because most are promptly executed.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
New Hypothesis: There are a lot of people on death row, and thousands get killed without anyone noticing.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Premise: Generally, if pH of scrubbing liquor falls below a range of 5.0 to 6.0, additional reagent is required to maintain the reactivity of the absorbent. Hypothesis: if pH of scrubbing liquor falls below a range of 5.0 to 6.0, then the whole world may explode (Contradiction) New Hypothesis: If the pH of scrubbing liquor falls below a range of 5.0 to 6.0 more reagent is needed to maintain it's reactivity New Hypothesis: if pH of scrubbing liquor falls below a range of 5.0 to 6.0, additional reagent will be needed to make the absorbent last longer
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Premise: If any of us at the dental school can be of assistance, please write or call. Hypothesis:
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
The dental school may or may not be able to help. (Contradiction) New Hypothesis: The dental school cannot give any help.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
New Hypothesis: We at the dental school at ready to help, please give us a call. Table 17: Qualitative differences between counterfactual edits made by humans when they are shown retrieved data vs. when they are not. Human annotators primed with retrieval are less likely to use trivial heuristics like negation, antonyms etc, leading to less bias introduced in the new data.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Counterfactual reasoning -mentally simulating what would have happened if conditions were different -is a common tool for making causality assessments (Kahneman and Tversky, 1981), which in turn are crucial for model evaluation, error analysis, and explanation (Miller, 2019). For example, in Figure 1, "It is great for kids" is perturbed into multiple variations, each providing unique insights by simulating what would have happened if the sentence was different.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Applications of counterfactual reasoning to NLP generally specify the relationship x )x, and then createx according to the relationship. As a result, prior work has tailored counterfactual generators for different applications, only collecting subsets ofx that are useful for the specific task. For example, to support model training and evaluation, human annotators create counterfactuals It is great for kids.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
It is great for kids→adults. It is great→scary for kids. 
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
It is not great for kids. It is great for kids→no one. that change the groundtruth labels by manually rewriting instances (Gardner et al., 2020;Qin et al., 2019) or defining perturbation functions (Ribeiro et al., 2020). Manual rewrites are costly (e.g., 4-5 minutes per counterfactual (Kaushik et al., 2020)) and susceptible to systematic omissions (e.g., human annotators may cover great ) not great, but miss kids ) no one in Figure 1B). Meanwhile, automated generators for model analysis and explanation usually focus on other relationships, e.g., generatingx that have different model predictions than x (Ross et al., 2020;Zhang et al., 2019a). As a result, they neglect prediction-preserving counterfactuals that are equally important for understanding or shaping model behaviors, like kids ) no one and great ) scary linked to Figure 1D. However, counterfactual generation does not have to be task-specific. The same set of counterfactuals in Figure 1 can support a variety of applica-tions. Moreover, for cases like model explanation and analysis, a general-purpose pool of counterfactuals may be preferable, as the relationship of interest can be more exploratory and user-oriented (Wu et al., 2019). In this work, we formalize the task of counterfactual generation, disentangling generation from the application of counterfactuals. Given an input x ( Figure 1A), our generator produces a set of counterfactualsX = {x 1 ,x 2 , ...} with applicationagnostic relationships x )x i ( Figure 1B). Afterwards, we use application-specific selection methods to find subsets ofx that are most effective for a given use case ( Figure 1C).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
It is not great for kids. It is great for kids→no one. that change the groundtruth labels by manually rewriting instances (Gardner et al., 2020;Qin et al., 2019) or defining perturbation functions (Ribeiro et al., 2020). Manual rewrites are costly (e.g., 4-5 minutes per counterfactual (Kaushik et al., 2020)) and susceptible to systematic omissions (e.g., human annotators may cover great ) not great, but miss kids ) no one in Figure 1B). Meanwhile, automated generators for model analysis and explanation usually focus on other relationships, e.g., generatingx that have different model predictions than x (Ross et al., 2020;Zhang et al., 2019a). As a result, they neglect prediction-preserving counterfactuals that are equally important for understanding or shaping model behaviors, like kids ) no one and great ) scary linked to Figure 1D. However, counterfactual generation does not have to be task-specific. The same set of counterfactuals in Figure 1 can support a variety of applica-tions. Moreover, for cases like model explanation and analysis, a general-purpose pool of counterfactuals may be preferable, as the relationship of interest can be more exploratory and user-oriented (Wu et al., 2019). In this work, we formalize the task of counterfactual generation, disentangling generation from the application of counterfactuals. Given an input x ( Figure 1A), our generator produces a set of counterfactualsX = {x 1 ,x 2 , ...} with applicationagnostic relationships x )x i ( Figure 1B). Afterwards, we use application-specific selection methods to find subsets ofx that are most effective for a given use case ( Figure 1C).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We frame the generation step as conditional text generation, and finetune GPT-2 (Radford et al., 2019) into a generator called Polyjuice using (x,x) pairs. To allow for targeted counterfactuals, we also design control codes like negation or delete ( Figure 1B), and adopt fill-in-the-blank structures (Donahue et al., 2020) to specify where the perturbation occurs and how. Intrinsic evaluation shows that Polyjuice generatesx that are fluent, diverse, and close to x, and that the control mechanisms retrieve perturbations that would likely not be sampled from off-the-shelf language models.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
With simple selection heuristics, we show that a single Polyjuice model can significantly aid humans in diverse downstream applications. 2 For counterfactual training and evaluation ( §3), humans label Polyjuice counterfactuals rather than creating them from scratch. They produce training data that significantly improve model generalization, as well as contrast sets that help identify model vulnerabilities (Gardner et al., 2020), with around 70% less annotation effort. In another application, Polyjuice produces counterfactual explanations ( §4), providing significant insight on top of state-of-the-art explanation techniques. Finally, Polyjuice supports counterfactual error analysis ( §5). It allows users to explore related counterfactuals (e.g., the model responds differently to different negation forms in Figure 1B), and to aggregate individual counterfactuals into patterns in order to gain systematic understanding of model behavior.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
2 General-Purpose Counterfactuals
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Given an instance x, a generator g produces a set of counterfactualsX = {x 1 ,x 2 , ...} with various re- 2 We demonstrate Polyjuice in semi-automatic settings, but as discussed in §2.2, it can also work automatically. Figure 2: (A) Polyjuice prompt format, which concatenates the original x, the control code, and thex ("It is not great for children" converted to an infilling structure). At generation time, Polyjuice accepts prompts that just include x (Line 1), or optionally with the code and the [BLANK]s (Lines 2-3), and fills in the blanks sequentially with spans separated by [ANSWER]s (Line 4). (B) Polyjuice allows blanking at different granularities (even the entire sentence), such that Lines 3-4 in (A) can be replaced by Lines 6-7 or 8-9. lationships x )x i . For example, great ) not great, kids ) no one in Figure 1B are both instances of the negation relationship. Each (x,x) pair shares multiple relationships -these two are also instances of the label flipping relationship if the task is sentiment analysis (but might not be for other tasks). As illustrated in §1, knowing which relationships apply aids selection for downstream applications.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We expect g to produce counterfactualsx that are (1) close to x, preferably only involving the minimal changes necessary to establish a certain effect (Pearl, 2018), allowing users to make causality assessments. The generatedx should also be (2) fluent, i.e., grammatically correct (Morris et al., 2020) and semantically meaningful (e.g.,"Colorless green ideas sleep furiously" is not meaningful (Chomsky, 2002)). Fluency operationalizes "probable" counterfactuals in the context of NLP; as Kahneman and Tversky (1981) stated, humans strongly favor counterfactuals that are close to the original instance, but also prefer those that could have easily happened without assuming rare events or strange coincidences. Further, as a general-purpose generator, g should produce counterfactuals with a measure of (3) control over relationships x )x, such that the counterfactuals can vary with the object-of-attention in each application (the "focus rule" (Kahneman and Tversky, 1981)). Finally, we expect g to output a (4) diverse set ofx in terms of relationships, covering a large variety of "what-ifs" for different applications (Pearl, 2018). shuffle To move (or swap) key phrases or entities around the sentence. A dog ) woman is embraced by the woman ) dog. (Zhang et al., 2019b) lexical To change just one word or noun chunk without altering the POS tags. A dog is embraced ) attacked by the woman. (Sakaguchi et al., 2020) resemantic To replace short phrases without altering the remaining dependency tree. A dog is embraced by the woman ) wrapped in a blanket. (Wieting and Gimpel, 2018) insert To add short phrases without altering the remaining dependency tree. restructure To alter the dependency tree structure, e.g., changing from passive to active. A dog is embraced by ) hugging the woman. (Wieting and Gimpel, 2018) 
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We expect g to produce counterfactualsx that are (1) close to x, preferably only involving the minimal changes necessary to establish a certain effect (Pearl, 2018), allowing users to make causality assessments. The generatedx should also be (2) fluent, i.e., grammatically correct (Morris et al., 2020) and semantically meaningful (e.g.,"Colorless green ideas sleep furiously" is not meaningful (Chomsky, 2002)). Fluency operationalizes "probable" counterfactuals in the context of NLP; as Kahneman and Tversky (1981) stated, humans strongly favor counterfactuals that are close to the original instance, but also prefer those that could have easily happened without assuming rare events or strange coincidences. Further, as a general-purpose generator, g should produce counterfactuals with a measure of (3) control over relationships x )x, such that the counterfactuals can vary with the object-of-attention in each application (the "focus rule" (Kahneman and Tversky, 1981)). Finally, we expect g to output a (4) diverse set ofx in terms of relationships, covering a large variety of "what-ifs" for different applications (Pearl, 2018). shuffle To move (or swap) key phrases or entities around the sentence. A dog ) woman is embraced by the woman ) dog. (Zhang et al., 2019b) lexical To change just one word or noun chunk without altering the POS tags. A dog is embraced ) attacked by the woman. (Sakaguchi et al., 2020) resemantic To replace short phrases without altering the remaining dependency tree. A dog is embraced by the woman ) wrapped in a blanket. (Wieting and Gimpel, 2018) insert To add short phrases without altering the remaining dependency tree. restructure To alter the dependency tree structure, e.g., changing from passive to active. A dog is embraced by ) hugging the woman. (Wieting and Gimpel, 2018) 
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We frame counterfactual generation as a conditional text generation task using language models (LMs), and train Polyjuice by finetuning GPT-2 (Radford et al., 2019) using the following prompt design (alternative LMs could also have been used).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Prompt format design. To ensure thatx is close to x rather than arbitrary text, we condition the generation on x, followed by a special token (Line 1 in Figure 2A). In Line 2, we have control codes (Keskar et al., 2019) such as negation. We design them to specify types of perturbation from among lexical, syntactic, or semantic aspects (see Table  1), inspired by prior work that categorizes manually created counterfactuals (Kaushik et al., 2020;Gardner et al., 2020). As an additional layer of control over x )x, we allow users to specify where changes happen by having the LM infill [BLANK] tokens (Donahue et al., 2020), rather than generating arbitrary counterfactuals (Lines 3-4).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Finetuning GPT-2 -a causal LM for predicting next tokens -additionally allows us to exercise control at various levels of granularity. At generation time, if the user provides only the original example, Polyjuice will generate the control code, the blank locations, and the infilling (Lines 2-4). Alternatively, the user can specify the control code, or the control code and the blanks, to exercise different degrees of control depending on the application. As later shown in §4 and §5, such control is important for different use cases.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Training data. To train a conditional model, we combine six existing sentence-pair datasets, each containing a subset of the desired phenomena in Table 1. Further, we find naturally occurring sentence pairs (filtered by edit distance to guarantee closeness) in non-paired datasets including CommonGen (Lin et al., 2020), Natural Questions (Kwiatkowski et al., 2019, and SQuAD (Rajpurkar et al., 2016), such that the resulting dataset contains diverse counterfactuals. 3 We translate these sentence pairs into the format given in Figure 2A. For each (x,x), we compute its primary control code using part-of-speech tags and dependency trees. For example, negation occurs when we observe changes to negation modifiers or specific words like "supposedly", and shuffle occurs when we have overlap between tokens deleted and added. When multiple changes occur, we label it with the control code which most significantly changes the semantics of the corresponding subphrase as computed by SBERT (Reimers and Gurevych, 2019). For example, in Figure 2A, negation (great ) not great) is more significant than lexical (kids ) children). To balance the distribution ( Table 7 in Appendix A), for each dataset, we extract control codes from all the (x,x), 4 and randomly sample up to 10,000 instances per codes.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
In order to allow for flexible blanking at generation time, we generate multiple training prompts per pair, covering different dependency tree struc-  tures related to the perturbed spans ( Figure 2B), including (1) just the changed tokens, (2) the associated parsing structures, (3) the merged changes, and (4) the entire sentence. We eventually obtain 657, 144 prompts from 186, 451 pairs.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Fluency filtering. While the original GPT-2 produces fluent text, some combinations of control codes and blanks cause Polyjuice to generate nonsensical results. Following Morris et al. (2020), we score both x andx with GPT-2, and filterx when the log-probability (on the full sentence or the perturbed chunks) decreases by more than 10 points relative to x. Fully automated uses of Polyjuice (e.g., adversarial attacks) may benefit from stricter constraints, at the cost of diversity (as surprising changes may be filtered even if they are fluent).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We evaluate Polyjuice on closeness and diversity by comparing its perturbations on 300 randomly selected sentences with baselines that use more or less context from x: (1) non-finetuned GPT-2, (2) token-infilling RoBERTa (Liu et al., 2019) and (3) span-infilling T5 (Raffel et al., 2020). As shown in Table 2, Polyjuice generates counterfactuals that are close to the original instance, measured by syntactic tree (Zhang and Shasha, 1989) and Levenshtein edit distance (Levenshtein, 1966). In contrast, non-finetuned GPT-2 generates arbitrary text instead of perturbations when given the starting tokens of a sentence, as it only leverages context in a single direction. As for infilling models, Polyjuice counterfactuals are more diverse (measured by self-BLEU (Zhu et al., 2018)) than RoBERTa ones, which is restricted to word substitution. Meanwhile, T5 displays higher diversity but less closeness, probably due to the fact that it does not consider the original masked tokens when generatingx. For example, in Figure 1 "It is great for kids," T5 replaces "for kids" with "idea", "to meet you," whereas Polyjuice generates "for kids yet adults can enjoy," "for any audience."
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We evaluate Polyjuice on closeness and diversity by comparing its perturbations on 300 randomly selected sentences with baselines that use more or less context from x: (1) non-finetuned GPT-2, (2) token-infilling RoBERTa (Liu et al., 2019) and (3) span-infilling T5 (Raffel et al., 2020). As shown in Table 2, Polyjuice generates counterfactuals that are close to the original instance, measured by syntactic tree (Zhang and Shasha, 1989) and Levenshtein edit distance (Levenshtein, 1966). In contrast, non-finetuned GPT-2 generates arbitrary text instead of perturbations when given the starting tokens of a sentence, as it only leverages context in a single direction. As for infilling models, Polyjuice counterfactuals are more diverse (measured by self-BLEU (Zhu et al., 2018)) than RoBERTa ones, which is restricted to word substitution. Meanwhile, T5 displays higher diversity but less closeness, probably due to the fact that it does not consider the original masked tokens when generatingx. For example, in Figure 1 "It is great for kids," T5 replaces "for kids" with "idea", "to meet you," whereas Polyjuice generates "for kids yet adults can enjoy," "for any audience."
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We evaluate controllability by comparing Polyjuice with T5 as well as with GPT-2 finetuned on prompts without codes. We verify that the codes improve the success rate of generating counterfactuals with the desired perturbation types set out in Table 1 by as much as 42% for perturbations such as negation and insert. For example, given "It is [BLANK] great for kids," baselines generate "also," "fun and," rather than "not" (negation).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We further verify the fluency for Polyjuice counterfactuals in three tasks/datasets: (1) Sentiment Analysis, SST-2 (Socher et al., 2013), (2) Natural Language Inference (NLI), SNLI (Bowman et al., 2015), and (3) Duplicate Question Detection (QQP) (Wang et al., 2019). We randomly select 100 sentences per dataset, generate 3x per x, and ask crowd workers to rate whether they are "likely written by native speakers." The workers rated most counterfactuals as fluent: 78% in SST-2, 76% in QQP, and 86% in SNLI. In subsequent sections, we show these rates are suitable for applications where people "team up" with Polyjuice.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We further verify the fluency for Polyjuice counterfactuals in three tasks/datasets: (1) Sentiment Analysis, SST-2 (Socher et al., 2013), (2) Natural Language Inference (NLI), SNLI (Bowman et al., 2015), and (3) Duplicate Question Detection (QQP) (Wang et al., 2019). We randomly select 100 sentences per dataset, generate 3x per x, and ask crowd workers to rate whether they are "likely written by native speakers." The workers rated most counterfactuals as fluent: 78% in SST-2, 76% in QQP, and 86% in SNLI. In subsequent sections, we show these rates are suitable for applications where people "team up" with Polyjuice.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We ask crowdworkers to label Polyjuice-generated counterfactuals for Sentiment, NLI, and QQP, for the purposes of evaluation and training. 5 In each labeling round, the worker is presented with an original x and its label, and asked to annotate the groundtruth for threex, rejecting non-fluent ones (details and interface in Appendix B.1).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We use a simple heuristic to select which counterfactuals are presented for labeling, aimed at increasing diversity. Representing eachx by its token changes, control code, and dependency tree structure, we greedily select the ones that are least similar to those already selected for labeling. This avoids redundancy in the labeling set, e.g., common perturbation patterns such as black ) white.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We verify whether Polyjuice counterfactuals can be used to create contrast sets (Gardner et al., 2020), i.e., evaluation sets where each instance has a nearby counterfactual with a different groundtruth, to better evaluate model decision boundaries. We  construct these sets by simply filtering out counterfactuals that are labeled the same as their original instances (40%-63% depending on the task).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
For each task, we test multiple classifers opensourced by Huggingface (Wolf et al., 2020), and report the best performing model for each 6 in Table 3 (results for other models are analogous). Polyjuice contrast sets display performance gaps consistent with those of Gardner et al. (2020), where the sets are constructed manually by NLP researchers, even though we use non-expert annotators who only label examples rather than creating them. To distinguish the benefit of counterfactuals from that of just adding more data, we further add a baseline that uses n + m original examples (m-baseline). In addition to in-domain test set accuracy, we measure models' generalization on out-of-domain datasets, as well as contrast sets and challenge sets. We also evaluate model capabilities with CheckList (Ribeiro et al., 2020) for Sentiment and QQP. Reported model performances are averaged across multiple data samples and random seeds (Appendix B.2).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
For each task, we test multiple classifers opensourced by Huggingface (Wolf et al., 2020), and report the best performing model for each 6 in Table 3 (results for other models are analogous). Polyjuice contrast sets display performance gaps consistent with those of Gardner et al. (2020), where the sets are constructed manually by NLP researchers, even though we use non-expert annotators who only label examples rather than creating them. To distinguish the benefit of counterfactuals from that of just adding more data, we further add a baseline that uses n + m original examples (m-baseline). In addition to in-domain test set accuracy, we measure models' generalization on out-of-domain datasets, as well as contrast sets and challenge sets. We also evaluate model capabilities with CheckList (Ribeiro et al., 2020) for Sentiment and QQP. Reported model performances are averaged across multiple data samples and random seeds (Appendix B.2).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
For Sentiment, we select random Polyjuice counterfactuals regardless of their labels, as long as an original x has at least onex that flips the label. For NLI and QQP, we observed in a pilot study that randomly chosen counterfactuals may not be more effective than the same amount of additional data. We suspect that Polyjuice lacks domain knowledge and context for identifying critical perturbations, and therefore brings benefits redundant with pretraining (Longpre et al., 2020). Thus, we use the slicing functions of Chen et al. (2019) to find patterns of interest (e.g., prepositions in NLI), and perturb those patterns by placing [BLANK]s on the matched spans. For example, "His surfboard is beneath him" becomes "His surfboard is [BLANK] him", and Polyjuice generates counterfactuals such as "His surfboard is beneath ) next to him."
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Results. Tables 4-6 indicate that Polyjuice augmentation is effective in all tasks: m-polyjuice maintains in-domain accuracy while consistently improving or maintaining generalization accuracy in various out-of-domain and challenge sets. On NLI, Polyjuice counterfactuals are as effective or more effective than counterfactuals created from scratch (m-CAD). Notably, we obtain the largest gains on challenge and contrast sets (e.g., Break and DNC in Table 5) or when the out-of-domain dataset is sufficiently different from the training domain (e.g., Senti140 and SemEval in Table 4). Polyjuice also improves results on CheckList tests that previously had high error rates: it significantly lowers the error rates on 11 out of 27 QQP tests, 7 making 2/27 tests worse. For Sentiment, it improves the model on 5 out of 15 tests, hurting 1. Here, we only report a low m/n ratio (<10% for NLI and QQP) to show that a small amount of augmentation is already beneficial. The results are similar for other combinations we explored (see Appendix B.2), except when the ratio of counterfactual to original data was too high (e.g.,, m = n may decrease vocabulary diversity or induce additional data bias, echoing (Khashabi et al., 2020)).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We show that Polyjuice counterfactuals are useful for evaluation, and more effective than additional (non-counterfactual) data for training in a variety of tasks. In contrast to prior work where humans generate counterfactuals from scratch, we only ask them to label automatically generated ones, while still achieving similar or better results.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We believe our approach is more effective than manual creation (although both are beneficial): in     Table 6: Polyjuice with n=20, 000 and m=1, 911 improves accuracy on PAWS-QQP (Zhang et al., 2019b). terms of implementation effort, the process of just labeling counterfactuals is the same as labeling original examples, such that no additional annotator training or separate pipelines are required; in contrast, Kaushik et al. (2020) set up two separate crowdsourcing tasks for creating and labeling the counterfactuals. Further, annotator effort is much lower, as evaluating examples is easier than creating them -Kaushik et al. (2020) report an average of ≈2 minutes per NLI counterfactual prior to quality validation, while our median time was 10 seconds per counterfactual. Even after our quality validation (removing noisy annotators, disregarding non-fluent counterfactuals), our rate for NLI is ≈36 seconds per counterfactual (used in Table 5).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We believe our approach is more effective than manual creation (although both are beneficial): in     Table 6: Polyjuice with n=20, 000 and m=1, 911 improves accuracy on PAWS-QQP (Zhang et al., 2019b). terms of implementation effort, the process of just labeling counterfactuals is the same as labeling original examples, such that no additional annotator training or separate pipelines are required; in contrast, Kaushik et al. (2020) set up two separate crowdsourcing tasks for creating and labeling the counterfactuals. Further, annotator effort is much lower, as evaluating examples is easier than creating them -Kaushik et al. (2020) report an average of ≈2 minutes per NLI counterfactual prior to quality validation, while our median time was 10 seconds per counterfactual. Even after our quality validation (removing noisy annotators, disregarding non-fluent counterfactuals), our rate for NLI is ≈36 seconds per counterfactual (used in Table 5).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
In terms of the utility per counterfactual, manual creation and Polyjuice may be complementary. Manual annotation may be unreliable or incomplete for certain forms of counterfactuals (Ribeiro et al., 2018), whereas Polyjuice can miss more complex or context-dependent changes, and could benefit from target perturbations that compensate for its lack of domain knowledge (targeted guidance is also helpful for human annotators (Huang et al., 2020)). Thus, it may be important to mix both approaches (Khashabi et al., 2020). Polyjuice's flexibility opens up possibilities for hybrids between human creation and human verification of targeted, machine-generated counterfactuals. , perturbed Q2
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
In terms of the utility per counterfactual, manual creation and Polyjuice may be complementary. Manual annotation may be unreliable or incomplete for certain forms of counterfactuals (Ribeiro et al., 2018), whereas Polyjuice can miss more complex or context-dependent changes, and could benefit from target perturbations that compensate for its lack of domain knowledge (targeted guidance is also helpful for human annotators (Huang et al., 2020)). Thus, it may be important to mix both approaches (Khashabi et al., 2020). Polyjuice's flexibility opens up possibilities for hybrids between human creation and human verification of targeted, machine-generated counterfactuals. , perturbed Q2
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
is Duplicate (=) at 98.2% confidence, with SHAP importance weights for tokens in Q2. Counterfactual explanations complement SHAP with concrete examples and surprising behaviors, e.g., (B) shows that friend ) woman surprisingly flips the prediction to Non-Duplicate ( ), despite the low weight on "friend."
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
A popular way of explaining NLP models is to attribute importance weights to the input tokens, either using attention scores (Wiegreffe and Pinter, 2019) or by summarizing the model behavior on perturbed instances (e.g., LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017)). Though ubiquitous, token scores may not always reflect their real importance (Pruthi et al., 2020). Popular packages like LIME or SHAP estimate scores by masking words, and therefore may not reflect model behavior on natural counterfactual cases. For example, the token "friend" in Figure 3A is not considered important even though a natural substitution in Figure 3B flips the prediction. The opposite happens to "in depression," where a significant change makes no difference to the model's prediction ( Figure 3C). Even perfect importance scores may be too abstract for users to gain real understanding (Miller, 2019), e.g., users may not grasp the significance of a low importance score for the token "help" without concrete examples such as the one in Figure 3D.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
A popular way of explaining NLP models is to attribute importance weights to the input tokens, either using attention scores (Wiegreffe and Pinter, 2019) or by summarizing the model behavior on perturbed instances (e.g., LIME (Ribeiro et al., 2016) and SHAP (Lundberg and Lee, 2017)). Though ubiquitous, token scores may not always reflect their real importance (Pruthi et al., 2020). Popular packages like LIME or SHAP estimate scores by masking words, and therefore may not reflect model behavior on natural counterfactual cases. For example, the token "friend" in Figure 3A is not considered important even though a natural substitution in Figure 3B flips the prediction. The opposite happens to "in depression," where a significant change makes no difference to the model's prediction ( Figure 3C). Even perfect importance scores may be too abstract for users to gain real understanding (Miller, 2019), e.g., users may not grasp the significance of a low importance score for the token "help" without concrete examples such as the one in Figure 3D.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Since presenting a large number of concrete counterfactuals would be overwhelming, we propose a hybrid approach, displaying feature attributions as a high-level summary, together with a judicious selection of Polyjuice counterfactuals that make behaviors concrete and highlight potential limitations. Following Miller (2019)'s observation that people look for explanations revealing unexpected behavior, we select surprising counterfactuals. 8 That is, we estimate the expected change in prediction with feature attributions, and select counterfactuals that violate these expectations, i.e., examples where the real change in prediction is large even though importance scores are low (Figure 3B), and examples where the change is small but importance scores are high ( Figure 3C). Of course, users can also view additional counterfactuals that perturb tokens of particular interest, a technique that we explore in the next section.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
User evaluation. We study the scenario where an expert has access to a model and local explanations, and evaluate the additional benefit of showing counterfactuals, i.e., whether they bring new insights. We evaluate three ways of generating counterfactuals: (1) Polyjuice-random, a baseline where we show random Polyjuice counterfactuals, (2) Expert-surprise, where two graduate students (non-participants) were given access to the model and instructed to create counterfactuals that are surprising given the associated SHAP scores, and (3) Polyjuice-surprise, which uses the selection procedure described in the previous paragraph.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We recruited 13 participants (graduate students with experience in model explanation), and had them analyze the aforementioned QQP model. In each round, users were shown an example, the model prediction, and a SHAP explanation, as in Figure 3A. Users were instructed to create up to 10 counterfactuals in order to better understand model behavior around the example, for which model predictions were given (users created 6 on average). Finally, users simulated what the model would do on six counterfactuals (Hase and Bansal, 2020), two from each condition (in random order). Counterfactuals where users make mistakes are prefer-8 Details in Appendix C.1.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
20% 30% 40% 50%
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Polyjuice-random
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Polyjuice-surprise Figure 4: Simulation error rates per condition (higher the better). Polyjuice-surprise has the highest error rate, indicating these counterfactuals would add the most information to users if displayed.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
able, as displaying these would add information that users do not already have. As shown in Figure 4, humans simulated model behavior on Polyjuice-surprise counterfactuals only slightly better than random guessing (45% ± 6%), i.e., these examples display model behavior that is surprising to users even after seeing explanations and creating their own counterfactuals. Expert-surprise also had a high error rate, but at a much higher cost: generating these for just 20 original instances took 1.5-2 hours of expert labor.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
While high error rates could be achieved with unrelated or nonsensical examples, all counterfactuals under evaluation were close to the original examples, when measured by syntactic tree edit (≈1.0) or Levenshtein distance (≈0.2), Polyjuicesurprise being the closest on both. An independent rater labeled 95% of Polyjuice-surprise counterfactuals as "likely written by a native speaker," in contrast to 85% for Expert-surprise, indicating that experts sometimes resorted to ungrammatical or nonsensical sentences to find surprising behaviors.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Qualitatively, the study participants tended to create counterfactuals by perturbing the token with the highest weights (84% of theirx perturbed tokens in the top 15% quantile of weights), not gaining a real understanding of how the other tokens impact predictions. Participants also made a significant number of mistakes even for tokens they had inspected, e.g., a participant perturbed the example in Figure 3A by replacing help ) play with, yielding a Non-Duplicate model prediction. When faced with help ) find in Figure 3D, they incorrectly assumed the behavior would be the same.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
These results indicate that Polyjuice counterfactuals complement feature attribution explanations by displaying information that users often miss, even after they have manually explored the model behavior beyond explanations. Moreover, Polyjuice counterfactuals for this application were more surprising and fluent than Expert-surprise, despite being computed automatically.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
, perturbed H through [negation] x P: A woman is holding a baby by a window. H: This woman is looking out the window.  
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
While our use of Polyjuice has so far relied on automatic selection of counterfactuals, we show in this section how an analyst can benefit from multiple counterfactuals per x, make use of controlled generation for more advanced analysis, and extract general patterns from individual observations. Our use case is counterfactual error analysis (Wu et al., 2019) of RoBERTa finetuned on NLI (used in §3.1), although the techniques are generally applicable.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
While our use of Polyjuice has so far relied on automatic selection of counterfactuals, we show in this section how an analyst can benefit from multiple counterfactuals per x, make use of controlled generation for more advanced analysis, and extract general patterns from individual observations. Our use case is counterfactual error analysis (Wu et al., 2019) of RoBERTa finetuned on NLI (used in §3.1), although the techniques are generally applicable.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
There is a known correlation between the label Contradiction and hypotheses with negation in NLI datasets (Gururangan et al., 2018), which may cause models to fail on non-contradiction negations. We explore this in Figure 5A by generating counterfactual hypotheses for a random Neutral instance, conditioning only on the original x and the negation control code. While the first two counterfactuals display this failure mode, there is a surprising inconsistency in model behavior between "not" and "n't". We note that manual analysis may not explore these three negation forms, and thus not surface this puzzling behavior.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
To verify if the pattern is widespread, we generate counterfactuals with the negation control code for a random set of instances correctly predicted as Neutral (n = 895). To generalize individual changes into patterns, we extract frequent counterfactual templates with Tempura  (details in Appendix D.2), shown in Figure 5B  its prediction from Neutral to Contradiction with roughly the same frequency (≈43%) whether the negation word is "not" or "n't", but flips much more frequently with a different negation pattern where a determiner is replaced with "no" (92.8%). While these behaviors may be correct in some instances, they often are not (e.g., Figure 5A), and thus would warrant further exploration, and potential mitigation strategies (e.g., counterfactual training, §3). Tangentially, the impact of DET ) no might lead the analyst to explore the impact of perturbing the subject of hypotheses, which we do in Figure 6 by placing a [BLANK] on the subject rather than using a control code. This leads to the discovery of unstable and erroneous behaviors regarding quantifiers, which we analyze in more detail in Appendix D.1.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Discussion. Polyjuice is a powerful tool for interactive analysis. Generating multiple counterfactuals per instance leads to insights that might be missed by manual analysis, and the steering provided by control codes and [BLANK]s allow for analyses that would be non-trivial to do manually (Wu et al., 2019) or with masked language models (e.g., Figure 5B places negations in various parts of sentences, and Figure 6 replaces spans with other spans of varying lengths). Besides error analysis, an analogous interactive use of Polyjuice may be suitable for test creation (Ribeiro et al., 2020) and forms of data augmentation that are more controlled than what we presented in §3.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Discussion. Polyjuice is a powerful tool for interactive analysis. Generating multiple counterfactuals per instance leads to insights that might be missed by manual analysis, and the steering provided by control codes and [BLANK]s allow for analyses that would be non-trivial to do manually (Wu et al., 2019) or with masked language models (e.g., Figure 5B places negations in various parts of sentences, and Figure 6 replaces spans with other spans of varying lengths). Besides error analysis, an analogous interactive use of Polyjuice may be suitable for test creation (Ribeiro et al., 2020) and forms of data augmentation that are more controlled than what we presented in §3.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We propose Polyjuice, a general-purpose generator that produces fluent and diverse counterfactuals, allowing for control over the kinds and locations of perturbations. With simple, task-specific selection heuristics, Polyjuice supports various downstream tasks on different domains, including counterfactual data augmentation, contrast set generation, counterfactual explanation, and error analysis.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
While Polyjuice is broadly applicable, it is not bias-free: control codes are pre-defined and certainly not exhaustive, and the model is fine-tuned on a collection of paired datasets where certain perturbations are more or less likely (e.g., we observe that words with negative sentiment tend to be slightly more likely than positive ones in some contexts). Collecting naturally occurring counterfactuals is an important area of future research, as is the development of generators that allow for control even without a-priori control codes.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Besides improving the generators, further work is needed to improve the value of counterfactuals. For example, while Polyjuice shows consistent gains across tasks in data augmentation, the improvements on some datasets are not as significant. This aligns with observations in prior work that even manual counterfactuals can be marginally beneficial (Kaushik et al., 2020;Huang et al., 2020), possibly because the original data is already diverse enough, or the perturbed signal in counterfactuals is too subtle to affect the model (e.g., when only a single word is changed in a long sentence.) We hope to perform more thorough experiments on tuning the amount and the distribution of counterfactual augmentation, as well as other ways of incorporating counterfactuals, such as having explicit terms in the loss function for contrasting counterfactuals with original data (Teney et al., 2020), or other forms of contrastive learning.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Besides improving the generators, further work is needed to improve the value of counterfactuals. For example, while Polyjuice shows consistent gains across tasks in data augmentation, the improvements on some datasets are not as significant. This aligns with observations in prior work that even manual counterfactuals can be marginally beneficial (Kaushik et al., 2020;Huang et al., 2020), possibly because the original data is already diverse enough, or the perturbed signal in counterfactuals is too subtle to affect the model (e.g., when only a single word is changed in a long sentence.) We hope to perform more thorough experiments on tuning the amount and the distribution of counterfactual augmentation, as well as other ways of incorporating counterfactuals, such as having explicit terms in the loss function for contrasting counterfactuals with original data (Teney et al., 2020), or other forms of contrastive learning.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Although our applications all involved people, the human-Polyjuice collaboration in labeling and explanations could benefit from richer interaction mechanisms. We believe Polyjuice motivates future research on more expressive forms of counterfactual training, where users generate counterfactuals together with Polyjuice, and label counterfactual patterns rather than individual instances. Similarly, interactive explanations and analysis are exciting directions, especially as we develop new ways of selecting, presenting, and aggregating counterfactuals for various analysis objectives. Having noted these opportunities, we believe Polyjuice is already a powerful tool for counterfactual reasoning, in particular for tasks where people are directly involved. Polyjuice is opensource, and available at https://github.com/tongshuangwu/polyjuice.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Our work includes labeling counterfactuals on crowdsourcing platforms, as well as conducting user studies with graduate students. As detailed in Appendix B.1 and C.2, we compensated the MTurk workers $2.5 for ≈15 minutes of labeling, and the graduate students $20 for the user study (one hour), above the U.S. federal minimum wage. The studies are with IRB approval.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We only finetune GPT-2 rather than training it from scratch, such that our compute costs are relatively low (around 8 hours for finetuning, Appendix A). All of our finetuning experiments involved finetuning RoBERTa on smaller datasets.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
More critically, with most of our demonstrated applications using a human-generator hybrid mechanism, we stress that the interaction between the two deserves careful consideration. It has long been reported that algorithms interacting with humans can negatively impact the human. 9 In our case, the concern might be that users can develop an over-reliance on Polyjuice (Bansal et al., 2021) and hastily accept its generations. Not only can this decrease users' creativity (Green et al., 2014), but it may bias their analysis process: as discussed in §7, Polyjuice generation is not exhaustive, and may favor some perturbation patterns over others in unpredictable ways. In the short term, we plan to highlight these limitations as part of the model documentation, while future research should identify interaction mechanisms, so as to ensure that Polyjuice or other counterfactual generators support humans, rather than hindering their performance.  A GPT-2 as Counterfactual Generator
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
More critically, with most of our demonstrated applications using a human-generator hybrid mechanism, we stress that the interaction between the two deserves careful consideration. It has long been reported that algorithms interacting with humans can negatively impact the human. 9 In our case, the concern might be that users can develop an over-reliance on Polyjuice (Bansal et al., 2021) and hastily accept its generations. Not only can this decrease users' creativity (Green et al., 2014), but it may bias their analysis process: as discussed in §7, Polyjuice generation is not exhaustive, and may favor some perturbation patterns over others in unpredictable ways. In the short term, we plan to highlight these limitations as part of the model documentation, while future research should identify interaction mechanisms, so as to ensure that Polyjuice or other counterfactual generators support humans, rather than hindering their performance.  A GPT-2 as Counterfactual Generator
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We combine several datasets to finetune Polyjuice. Contrast set. Authors of 10 existing NLP dataset each manually perturbed 100-1,000 instances to change the gold label, so to inspect a model's local decision boundary (Gardner et al., 2020). The perturbation patterns vary based on the tasks and the annotators, allowing us to learn diverse strategies. To make sure we can use the contrast set to evaluate the Sentiment model, we excluded the IMDb movie review from the training.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Counterfactually-augmented data (CAD). Kaushik et al. (2020) crowdsourced counterfactuals for IMDb movie review (1.7k), which we split into paired sentences to match the text length of other datasets. CAD's perturbation patterns also vary based on the task, but can especially contribute to negation. As NLI is in our demonstrating applications, we did not use their 6.6k SNLI counterfactuals. 10 WinoGrande is a large-scale dataset of 44k instances for testing common sense problems (Sakaguchi et al., 2020). It contains sentences that differ only by one trigger word (e.g., one noun), making it most suitable for learning lexical exchanges.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Counterfactually-augmented data (CAD). Kaushik et al. (2020) crowdsourced counterfactuals for IMDb movie review (1.7k), which we split into paired sentences to match the text length of other datasets. CAD's perturbation patterns also vary based on the task, but can especially contribute to negation. As NLI is in our demonstrating applications, we did not use their 6.6k SNLI counterfactuals. 10 WinoGrande is a large-scale dataset of 44k instances for testing common sense problems (Sakaguchi et al., 2020). It contains sentences that differ only by one trigger word (e.g., one noun), making it most suitable for learning lexical exchanges.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
ParaNMT-50M contains 50 million English-English sentential paraphrase pairs, covering various domains and styles of text, as well as different sentence structures (Wieting and Gimpel, 2018).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
ParaNMT-50M contains 50 million English-English sentential paraphrase pairs, covering various domains and styles of text, as well as different sentence structures (Wieting and Gimpel, 2018).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
PAWS (Zhang et al., 2019b) contains pairs with high text overlaps, created through controlled word swapping, best demonstrating shuffle and restructure. We used its 49k Wikipedia parts.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
PAWS (Zhang et al., 2019b) contains pairs with high text overlaps, created through controlled word swapping, best demonstrating shuffle and restructure. We used its 49k Wikipedia parts.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
HANS (McCoy et al., 2019), a challenge set for NLI, contains 10k pairs of premises and hypotheses created based on 10 heavily fallible syntactic templates, and therefore compensates rarer structural changes that may be missed by PAWS.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Crawled We additionally crawl naturally occurring sentence pairs from non-paired datasets boost some specific patterns and increase lexical diversity. This include (1)  , whose paragraphs involve Wikipedia knowledge. We estimate close pairs using edit distance, and broadly accept those with less than 60% editing. To exclude tricky cases (e.g.,"how do I not be" can be incorrectly regarded as negation for "how do I recover it"), we only augment the most determined patterns: lexical, insert, delete, and shuffle.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
To balance the distribution (Table 7), for each dataset, we extract control codes from all the (x,x), and randomly sample up to 10,000 instances per codes. Still, quantifier and negation have less training data compared to other codes. Fortunately, these codes tend to be limited to more specific patterns ("more than", "not", "never") when compared to "broad" codes like lexical, and thus even a small sample is enough to learn them. We finetuned an off-the-shelf GPT-2 model from Wolf et al. (2020) for 10 epochs with an initial learning rate 5e-5, a batch size of 8, and a sequence length of 120 (but any LM can potentially be used). We select the best epoch based on the evaluation loss on a holdout set of size 5,000. The training took around 8 hours on two Titan RTXs.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
To balance the distribution (Table 7), for each dataset, we extract control codes from all the (x,x), and randomly sample up to 10,000 instances per codes. Still, quantifier and negation have less training data compared to other codes. Fortunately, these codes tend to be limited to more specific patterns ("more than", "not", "never") when compared to "broad" codes like lexical, and thus even a small sample is enough to learn them. We finetuned an off-the-shelf GPT-2 model from Wolf et al. (2020) for 10 epochs with an initial learning rate 5e-5, a batch size of 8, and a sequence length of 120 (but any LM can potentially be used). We select the best epoch based on the evaluation loss on a holdout set of size 5,000. The training took around 8 hours on two Titan RTXs.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Similar to Madaan et al. (2021), we compare the diversity and closeness of Polyjuice with alternative generators, i.e., RoBERTa and T5, representing masked language models that prioritize word and span substitution, and original GPT-2, representing the standard generative model not conditioned on x. For a given x and its counterfactualsX, we approx-imate diversity using self-BLEU (Zhu et al., 2018) withinX. Meanwhile, closeness is the average distance between x and everyx ∈X, both with the normalized word level Levenshtein edit distance ((Levenshtein, 1966), used in MiCE (Ross et al., 2020)), and syntactic tree edit distance ( (Zhang and Shasha, 1989) in GYC (Madaan et al., 2021)).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Similar to Madaan et al. (2021), we compare the diversity and closeness of Polyjuice with alternative generators, i.e., RoBERTa and T5, representing masked language models that prioritize word and span substitution, and original GPT-2, representing the standard generative model not conditioned on x. For a given x and its counterfactualsX, we approx-imate diversity using self-BLEU (Zhu et al., 2018) withinX. Meanwhile, closeness is the average distance between x and everyx ∈X, both with the normalized word level Levenshtein edit distance ((Levenshtein, 1966), used in MiCE (Ross et al., 2020)), and syntactic tree edit distance ( (Zhang and Shasha, 1989) in GYC (Madaan et al., 2021)).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We run the three generators on 300 sentences in total. In GPT-2, we take the first two words of an x as the input context (prompt), limit the length of the generation to be similar to x, and collect 10 counterfactuals. As for RoBERTa and T5, we repeatedly perturb x for three times, each time randomly placing up to three [MASK] tokens, and ask the generator to generate 5 counterfactuals through beam search, following Ribeiro et al. (2020). Polyjuice uses the same blank (mask) placement as in RoBERTa and T5, but we additionally enumerate through all control codes. For each x, we randomly sample 5 counterfactuals to formX per generator.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We run the three generators on 300 sentences in total. In GPT-2, we take the first two words of an x as the input context (prompt), limit the length of the generation to be similar to x, and collect 10 counterfactuals. As for RoBERTa and T5, we repeatedly perturb x for three times, each time randomly placing up to three [MASK] tokens, and ask the generator to generate 5 counterfactuals through beam search, following Ribeiro et al. (2020). Polyjuice uses the same blank (mask) placement as in RoBERTa and T5, but we additionally enumerate through all control codes. For each x, we randomly sample 5 counterfactuals to formX per generator.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
As shown in Table 2, Polyjuice achieves a balance between diversity and closeness. Ideally, we would also like to compare Polyjuice with concurrent work (Madaan et al., 2021;Ross et al., 2020), but these are yet to be open-sourced and require extensive implementation or finetuning.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
To evaluate controllability, we compare Polyjuice with T5, and GPT-2 finetuned on prompts without codes (called Polyjuice -a), such that both baselines consider sufficient context. For each control code, we compare the control success rate of Polyjuice and Polyjuice-a on 300 prompts. For each prompt, we generate counterfactuals through beam search (beam = 5), and recompute the codes on the top three generatedx. We deem the control successful if at least one of the three recomputed codes matches the desired control code (though in Polyjuice-a, we only measure whether the code naturally occurs in the uncontrolled generation.) The success rate increases by 26% ± 13% across all control codes, ranging from quantifier (increasing 6%, from 50% to 56%) to negation (42%, from 5% to 47%). Non-finetuned T5 also achieves less control (success rate decreases by 33% on average.) Common failure cases include (1) The control codes conflict with the blanks, e.g.,"a dog is embraced by a [BLANK]" would not respond to negation.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
(2) x does not have a corresponding pattern, e.g., shuffle is not applicable to "the movie is good." (3) certain salient patterns dominate the generation probability, e.g., the model tends to perturb the quantifier "two" in "two dogs are running," regardless of the code.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
B.1 MTurk Labeling Details Procedure The study started with an introduction that explained the context and tasks. To familiarize crowdworkers with the task, we asked them to complete 1-2 training rounds, and explained the expected labels. Each annotator then completed 22 tasks, labeling 3 counterfactuals of a single example in each round, as in Figure 7. The 22 rounds consisted of 20 actual labeling tasks and 2 extra "gold rounds" with known correct labels. The gold cases later served to filter low-quality crowdworkers. The median annotation time was around 15 minutes, and participants received $2.5.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Participants. We recruited participants from MTurk, limiting the pool to subjects from within the US with a prior task approval rating of at least 97% and a minimum of 1,000 approved tasks.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Data quality. We applied two filtering strategies: (1) High-quality worker. We only kept data from participants whose median labeling time per round was more than 18 seconds and correctly labeled at least 4 gold counterfactuals (out of 6), or who correctly labeled all gold ones. (2) Majority vote labeling. We collected two annotations per counterfactual, and only kept those that at least one annotator deemed valid, and both annotators agreed on a particular class label. One of the authors la-4,000 4,500 5,000 5,500 6,000  Figure 8: The accuracy trend on two Sentiment datasets, as the total training datasize (m+n) varies. The blue line shows an augmentation of m = 2k counterfactuals, and the blue one represents the corresponding m-baseline.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Though the counterfactuals remains useful on datasets like SemEval across all m+n, it appears too many counterfactuals may be harmful (Amzbook).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
beled a subset of 100x on 100 x in Sentiment, and reached high agreement with the majority-voted results (κ = 0.77, raw labeling agreement 88%).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
For each (m, n), we created three samples of training data. Each sample was further averaged over four random seeds. For each run, we heuristically picked the initial learning rates 1e-5, 2e-5, 2e-5 for Sentiment, NLI and QQP, and trained 20 epochs with a dropout rate of 0.1 and a batch size of 16. We selected the epoch that had the highest accuracy on the corresponding validation set, which takes 1/5 of the training data size, with the same ratio of m/n counterfactual and original examples. We further explore ratios of added counterfactuals. Take Sentiment as an example: while the counterfactual remains effective on most datasets, it hurts the model performance on Amzbook when the counterfactual takes a large proportion (Figure 8, Yelp followed a similar but more mild trend). We suspect that flipping out too much original data affects the data diversity, and in turn decreases the model performance. Similarly, Huang et al. (2020) asserted that augmenting n = 1.7k NLI data with m = 6.6k counterfactuals did not improve model generalization accuracy.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Because SHAP weights reflect the average effect of masking a token t, we also focus on word features that are abnormal on average.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
More concretely, we define the expected changein-prediction for perturbing a token t to be the SHAP importance on it, H[D f (t, x)] = s(t). In Figure 3, s(t=depression) = 0.276. The actual prediction change D f (t, x) is the weighted average of | f p (x) − f p (x)| for all thex that affect t (depression ) trouble, depression ) a mood), where f p (x) is the prediction probability of f on x. The weight corresponds to the number of words modified inx: If e(x) denotes the set of edited words in x, then w(x) = 1/|e(x)|. Intuitively, the more words changed inx, the less impact each word has; In Figure 3D, we regard "depression" to be responsible for half of the impact in in depression ) suicidal. We groupx based on their affected words G t = {x | t ∈ e(x)}. D f (t, x) then becomes:
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
The additional SHAP weight s(t) acts as a smoothing factor to penalize outliers. Then the gap between the expectation and reality is:
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
We first find the abnormal tokens: (1) t with small SHAP weight, butx that change t experience large prediction change on average: t L = arg max t∈x ∆ D f (t, x), and (2) t with large SHAP weight, butx with t changed usually have intact prediction: t U = arg max t∈x −∆ D f (t, x).
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Then, we use the most extreme cases within the groups of G t L and G t U as the concrete counterfactual explanations, based on their prediction change | f p (x) − f p (x)|, and the aggregated SHAP weights of all the changed tokens: Figure 9 shows the sample interface. Participants started by just seeing the reference example and the model query box on the left hand side. When they chose to start the task or after they had exhausted their ten query chances, the query box was disabled, the tasks on the right were displayed, and the participants completed the tasks. We compensated participants $20 for the one hour study.
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
As a follow-up to Figure 6, we slice the data to find entailment instances that have numbers in the hypothesis sentence, and perturb their quantifiers.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Cognitive science and philosophy research has shown that human explanations are contrastive (Miller, 2019): People explain why an observed event happened rather than some counterfactual event called the contrast case. This contrast case plays a key role in modulating what explanations are given. Consider Figure 1. When we seek an explanation of the model's prediction "by train," we seek it not in absolute terms, but in contrast to another possible prediction (i.e. "on foot"). Additionally, we tailor our explanation to this contrast case. For instance, we might explain why the prediction is "by train" and not "on foot" by saying that the writer discusses meeting Ann at the train station Figure 1: An example MICE edit for a multiple-choice question from the RACE dataset. MICE generates contrastive explanations in the form of edits to inputs that change model predictions to target (contrast) predictions. The edit (bolded in red) is minimal and fluent, and it changes the model's prediction from "by train" to the contrast prediction "on foot" (highlighted in gray).
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
instead of at Ann's home on foot; such information is captured by the edit (bolded red) that results in the new model prediction "on foot." For a different contrast prediction, such as "by car," we would provide a different explanation. In this work, we propose to give contrastive explanations of model predictions in the form of targeted minimal edits, as shown in Figure 1, that cause the model to change its original prediction to the contrast prediction.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Given the key role that contrastivity plays in human explanations, making model explanations contrastive could make them more user-centered and thus more useful for their intended purposes, such as debugging and exposing dataset biases (Ribera and Lapedriza, 2019)-purposes which require that humans work with explanations (Alvarez-Melis et al., 2019). However, many currently popular instance-based explanation methods produce highlights-segments of input that support a prediction (Zaidan et al., 2007;Lei et al., 2016;Chang et al., 2019;Bastings et al., 2019;Yu et al., 2019;DeYoung et al., 2020;Jain et al., 2020;Belinkov and Glass, 2019) that can be derived through gradients (Simonyan et al., 2014;Smilkov et al., 2017;Sundararajan et al., 2017), approximations with simpler models (Ribeiro et al., 2016), or attention (Wiegreffe and Pinter, 2019;Sun and Marasović, 2021). These methods are not contrastive, as they leave the contrast case undetermined; they do not tell us what would have to be different for a model to have predicted a particular contrast label. 1 As an alternative approach to NLP model explanation, we introduce MINIMAL CONTRASTIVE EDITING (MICE)-a two-stage approach to generating contrastive explanations in the form of targeted minimal edits (as shown in Figure 1). Given an input, a fixed PREDICTOR model, and a contrast prediction, MICE generates edits to the input that change the PREDICTOR's output from the original prediction to the contrast prediction. We formally define our edits and describe our approach in §2.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We design MICE to produce edits with properties motivated by human contrastive explanations. First, we desire edits to be minimal, altering only small portions of input, a property which has been argued to make explanations more intelligible (Alvarez-Melis et al., 2019;Miller, 2019). Second, MICE edits should be fluent, resulting in text natural for the domain and ensuring that any changes in model predictions are not driven by inputs falling out of distribution of naturally occurring text. Our experiments ( §3) on three English-language datasets, IMDB, NEWSGROUPS, and RACE, validate that MICE edits are indeed contrastive, minimal, and fluent.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We also analyze the quality of MICE edits ( §4) and show how they may be used for two use cases in NLP system development. First, we show that MICE edits are comparable in size and fluency to human edits on the IMDB dataset. Next, we illustrate how MICE edits can facilitate debugging individual model predictions. Finally, we show how MICE edits can be used to uncover dataset artifacts learned by a powerful PREDICTOR model. 2 1 Free-text rationales (Narang et al., 2020) can be contrastive if human justifications are collected by asking "why... instead of..." which is not the case with current benchmarks (Camburu et al., 2018;Rajani et al., 2019;Zellers et al., 2019). 2 Our code and trained EDITOR models are publicly available at https://github.com/allenai/mice.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
This section describes our proposed method, MINI-MAL CONTRASTIVE EDITING, or MICE, for explaining NLP models with contrastive edits.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Contrastive explanations are answers to questions of the form Why p and not q? They explain why the observed event p happened instead of another event q, called the contrast case. 3 A long line of research in the cognitive sciences and philosophy has found that human explanations are contrastive (Van Fraassen, 1980;Lipton, 1990;Miller, 2019). Human contrastive explanations have several hallmark characteristics. First, they cite contrastive features: features that result in the contrast case when they are changed in a particular way (Chin-Parker and Cantelon, 2017). Second, they are minimal in the sense that they rarely cite the entire causal chain of a particular event, but select just a few relevant causes (Hilton, 2017). In this work, we argue that a minimal edit to a model input that causes the model output to change to the contrast case has both these properties and can function as an effective contrastive explanation. We first give an illustration of contrastive explanations humans might give and then show how minimal contrastive edits offer analogous contrastive information.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
As an example, suppose we want to explain why the answer to the question "Q: Where can you find a clean pillow case that is not in use?" is "A: the drawer." 4 If someone asks why the answer is not "C1: on the bed," we might explain: "E1: Because only the drawer stores pillow cases that are not in use." However, E1 would not be an explanation of why the answer is not "C2: in the laundry hamper," since both drawers and laundry hampers store pillow cases that are not in use. For contrast case C2, we might instead explain: "E2: Because only laundry hampers store pillow cases that are not clean." We cite different parts of the original question depending on the contrast case.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In this work, we propose to offer contrastive explanations in the form of minimal edits that result in the contrast case as model output. Such edits are effective contrastive explanations because, by construction, they highlight contrastive features. For Figure 2: An overview of MICE, our two-stage approach to generating edits. In Stage 1 ( §2.3), we train the EDITOR to make edits targeting specific predictions from the PREDICTOR. In Stage 2 ( §2.4), we make contrastive edits with the EDITOR model from Stage 1 such that the PREDICTOR changes its output to the contrast prediction. example, a contrastive edit of the original question for contrast case C1 would be: "Where can you find a clean pillow case that is not in use?"; the information provided by this edit-that it is whether or not the pillow case is in use that determines whether the answer is "the drawer" or "on the bed"-is analogous to the information provided by E1. Similarly, a contrastive edit for contrast case C2 that changed the question to "Where can you find a clean dirty pillow case that is not in use?" provides analogous information to E2.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We define a contrastive edit to be a modification of an input instance that causes a PREDIC-TOR model (whose behavior is being explained) to change its output from its original prediction for the unedited input to a given target (contrast) prediction. Formally, for textual inputs, given a fixed PREDICTOR f , input x = (x 1 , x 2 , ..., x N ) of N tokens, original prediction f (x) = y p and contrast prediction y c = y p , a contrastive edit is a mapping e : (x 1 , ...,
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We propose MICE, a two-stage approach to generating contrastive edits, illustrated in Figure 2. In Stage 1, we prepare a highly-contextualized EDI-TOR model to associate edits with given end-task labels (i.e., labels for the task of the PREDICTOR) such that the contrast label y c is not ignored in MICE's second stage. Intuitively, we do this by masking the spans of text that are "important" for the given target label (as measured by the PREDIC-TOR's gradients) and training our EDITOR to reconstruct these spans of text given the masked text and target label as input. In Stage 2 of MICE, we generate contrastive edits e(x) using the EDITOR model from Stage 1. Specifically, we generate candidate edits e(x) by masking different percentages of x and giving masked inputs with prepended contrast label y c to the EDITOR; we use binary search to find optimal masking percentages and beam search to keep track of candidate edits that result in the highest probability of the contrast labels p(y c |e(x)) given by the PREDICTOR.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In Stage 1 of MICE, we fine-tune the EDITOR to infill masked spans of text in a targeted manner. Specifically, we fine-tune a pretrained model to infill masked spans given masked text and a target end-task label as input. In this work, we use the TEXT-TO-TEXT TRANSFER TRANSFORMER (T5) model (Raffel et al., 2020) as our pretrained EDI-TOR, but any model suitable for span infilling can in principle be the EDITOR in MICE. The addition of the target label allows the highly-contextualized EDITOR to condition its predictions on both the masked context and the given target label such that the contrast label is not ignored in Stage 2. What to use as target labels during Stage 1 depends on who the end-users of MICE are. The end-user could be: (1) a model developer who has access to the labeled data used to train the predictor, or (2) layusers, domain experts, or other developers without access to the labeled data. In the former case, we could use the gold label as targets, and in the latter case, we could use the labels predicted by PREDIC-TOR. Therefore, during fine-tuning, we experiment with using both gold labels and original predictions y p of our PREDICTOR model as target labels. To provide target labels, we prepend them to inputs to the EDITOR. For more information about how these inputs are formatted, see Appendix B. Results in Table 2 show that fine-tuning with target labels results in better edits than fine-tuning without them.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
The above procedure allows our EDITOR to condition its infilled spans on both the context and the target label. But this still leaves open the question of where to mask our text. Intuitively, we want to mask the tokens that contribute most to the PREDICTOR's predictions, since these are the tokens that are most strongly associated with the target label. We propose to use gradient attribution (Simonyan et al., 2014) to choose tokens to mask. For each instance, we take the gradient of the predicted logit for the target label with respect to the embedding layers of f and take the 1 norm across the embedding dimension. We then mask the n 1 % of tokens with the highest gradient norms. We replace consecutive tokens (i.e., spans) with sentinel tokens, following Raffel et al. (2020). Results in Table 1 show that gradient-based masking outperforms random masking.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In the second stage of our approach, we use our finetuned EDITOR to make edits using beam search (Reddy, 1977). In each round of edits, we mask consecutive spans of n 2 % of tokens in the original input, prepend the contrast prediction to the masked input, and feed the resulting masked instance to the EDITOR; the EDITOR then generates m edits. The masking procedure during this stage is gradientbased as in Stage 1.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In one round of edits, we conduct a binary search with s levels over values of n 2 between values n 2 = 0% to n 2 = 55% to efficiently find a value of n 2 that is large enough to result in the contrast prediction while also modifying only minimal parts of the input. After each round of edits, we get f 's predictions on the edited inputs, order them by contrast prediction probabilities, and update the beam to store the top b edited instances. As soon as an edit e * = e(t) is found that results in the contrast prediction, i.e., f (e * ) = y c , we stop the search procedure and return this edit. For generation, we use a combination of top-k (Fan et al., 2018) and top-p (nucleus) sampling (Holtzman et al., 2020). 5
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
This section presents empirical findings that MICE produces minimal and fluent contrastive edits.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Tasks We evaluate MICE on three Englishlanguage datasets: IMDB, a binary sentiment classification task (Maas et al., 2011), a 6-class version of the 20 NEWSGROUPS topic classification task (Lang, 1995), and RACE, a multiple choice question-answering task (Lai et al., 2017). 6 PREDICTORS MICE can be used to make contrastive edits for any differentiable PREDICTOR model, i.e., any end-to-end neural model. In this paper, for each task, we train a PREDICTOR model f built on ROBERTA-LARGE (Liu et al., 2019), and fix it during evaluation. The test accuracies of our PREDICTORS are 95.9%, 85.3% and 84% for IMDB, NEWSGROUPS, and RACE, respectively. For training details, see Appendix A.1.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
EDITORS Our EDITORS build on the base version of T5. For fine-tuning our EDITORS (Stage 1), we use the original training data used to train PRE-DICTORS. We randomly split the data, 75%/25% for fine-tuning/validation and fine-tune until the validation loss stops decreasing (for a max of 10 epochs) with n 1 % of tokens masked, where n 1 is a randomly chosen value in [20,55]. For more details, see Appendix A.2. In Stage 2, for each instance, we set the label with the second highest predicted probability as the contrast prediction. We set beam width b = 3, consider s = 4 search levels during binary search over n 2 in each edit round, and run our search for a max of 3 edit rounds. For each n 2 , we sample m = 15 generations from our fine-tuned EDITORS with p = 0.95, k = 30. 7  Table 1: Efficacy of the MICE procedure. We evaluate MICE edits on three metrics (described in §3.1): flip rate, minimality, and fluency. We report mean values for minimality and fluency. * marks full MICE variants; others explore ablations. For each property (i.e., column), the best value across MICE variants is bolded. We experiment with PREDICTOR's predictions (PRED) and gold labels (GOLD) as target labels during Stage 1. Across datasets, our GRAD MICE procedure achieves a high flip rate with small and fluent edits.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
25K instances in the test set for evaluation because of the computational demands of evaluation. 9 For each dataset, we measure the following three properties: (1) flip rate: the proportion of instances for which an edit results in the contrast label; (2) minimality: the "size" of the edit as measured by the word-level Levenshtein distance between the original and edited input, which is the minimum number of deletions, insertions, or substitutions required to transform one into the other. We report a normalized version of this metric with a range from 0 to 1-the Levenshtein distance divided by the number of words in the original input; (3) fluency: a measure of how similarly distributed the edited output is to the original data. We evaluate fluency by comparing masked language modeling loss on both the original and edited inputs using a pretrained model. Specifically, given the original N -length sequence, we create N copies, each with a different token replaced by a mask token, following Salazar et al. (2020). We then take a pretrained T5-BASE model and compute the average loss across these N copies. We compute this loss value for both the original input and edited input and report their ratio-i.e., edited / original. We aim for a value of 1.0, which indicates equivalent losses for the original and edited texts. When MICE finds multiple edits, we report metrics for the edit with the smallest value for minimality.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Results are shown in Table 1. Our proposed GRAD MICE procedure (upper part of Table 1) achieves a 9 A single contrastive edit is expensive and takes an average of ≈ 15 seconds per IMDB instance (≈ 230 tokens). Calculating the fluency metric adds an additional average of ≈ 16.5 seconds per IMDB instance. For more details, see Section 5.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
high flip rate across all three tasks. This is the outcome regardless of whether predicted target labels (first row, 91.5-100% flip rate) or gold target labels (second row, 94.5-100% flip rate) are used for finetuning in Stage 1. We observe a slight improvement from using the gold labels for the RACE PREDIC-TOR, which may be explained by the fact that it is less accurate (with a training accuracy of 89.9%) than the IMDB and NEWSGROUPS classifiers.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
MICE achieves a high flip-rate while its edits remain small and result in fluent text. In particular, MICE on average changes 17.3-33.1% of the original tokens when predicted labels are used in Stage 1 and 18.5-33.5% with gold labels. Fluency is close to 1.0 indicating no notable change in mask language modeling loss after the edit-i.e., edits fall in distribution of the original data. We achieve the best results across metrics on the IMDB dataset, as expected since IMDB is a binary classification task with a small label space. These results demonstrate that MICE presents a promising research direction for the generation of contrastive explanations; however, there is still room for improvement, especially for more challenging tasks such as RACE.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In the rest of this section, we provide results from several ablation experiments.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Fine-tuning vs. No Fine-tuning We investigate the effect of fine-tuning (Stage 1) with a baseline that skips Stage 1 altogether. For this NO-FINETUNE baseline variant of MICE, we use the vanilla pretrained T5-BASE as our EDITOR. As shown in Table 1, the NO-FINETUNE variant underperforms all other (two-stage) variants of MICE for the IMDB and NEWSGROUPS datasets. 10 Fine- 10 We leave RACE out from our evaluation with the NO-FINETUNE baseline because we observe that the pretrained  Table 2: Effect of using target end-task labels during the two stages of PRED+GRAD MICE on the IMDB dataset. When end-task labels are provided, they are original PREDICTOR labels during Stage 1 and contrast labels during Stage 2. The best values for each property (column) are bolded. Using end-task labels during both Stage 1 (EDITOR fine-tuning) and Stage 2 (making edits) of MICE outperforms all other conditions. tuning particularly improves the minimality of edits, while leaving the flip rate high. We hypothesize that this effect is due to the effectiveness of Stage 2 of MICE at finding contrastive edits: Because we iteratively generate many candidate edits using beam search, we are likely to find a predictionflipping edit. Fine-tuning allows us to find such an edit at a lower masking percentage.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We study the impact of using gradient-based masking in Stage 1 of the MICE procedure with a RAND variant, which masks spans of randomly chosen tokens. As shown in the middle part of Table 1, gradient-based masking outperforms random masking when using both predicted and gold labels across all three tasks and metrics, suggesting that the gradient-based attribution used to mask text during Stage 1 of MICE is an important part of the procedure. The differences are especially notable for RACE, which is the most challenging task according to our metrics.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Targeted vs. Un-targeted Infilling We investigate the effect of using target labels in both stages of MICE by experimenting with removing target labels during Stage 1 (EDITOR fine-tuning) and Stage 2 (making edits). As shown in Table 2, we observe that giving target labels to our EDITORS during both stages of MICE improves edit quality. Fine-tuning EDITORS without labels in Stage 1 ("No Label") leads to worse flip rate, minimality, and fluency than does fine-tuning EDITORS with labels ("Label"). Minimality is particularly affected, and we hypothesize that using target end-task la-T5 model does not generate text formatted as span infills; we hypothesize that this model has not been trained to generate infills for masked inputs formatted as multiple choice inputs. bels in both stages provides signal that allows the EDITOR in Stage 2 to generate prediction-flipping edits at lower masking percentages.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In this section, we compare MICE edits with human contrastive edits. Then, we turn to a key motivation for this work: the potential for contrastive explanations to assist in NLP system development.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We show how MICE edits can be used to debug incorrect predictions and uncover dataset artifacts.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We ask whether the contrastive edits produced by MICE are minimal and fluent in a meaningful sense. In particular, we compare these two metrics for MICE edits and human contrastive edits. We work with the IMDB contrast set created by Gardner et al. (2020) 
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
When George was thirty-five, he bought a small plane and learned to fly it. He soon became very good and made his plane do all kinds of tricks. George had a friend, whose name was Mark. One day George offered to take Mark up in his plane. Mark thought, "I've traveled in a big plane several times, but I've never been in a small one, so I'll go." They went up, and George flew around for half an hour and did all kinds of tricks in the air. When they came down again, Mark was glad to be back safely, and he said to his friend in a shaking voice, "Well, George, thank you very much for those two trips tricks in your plane." George was very surprised and said, "Two trips? tricks." Yes, That's my first and my last time, George." answered said Mark. input in Table 3, for which the RACE PREDICTOR gives an incorrect prediction. In this case, a model developer may want to understand why the model got the answer wrong. This setting naturally brings rise to a contrastive question, i.e., Why did the model predict the wrong choice ("twice") instead of the correct one ("only once")?
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
The MICE edit shown offers insight into this question: Firstly, it highlights which part of the paragraph has an influence on the model prediction-the last few sentences. Secondly, it reveals that a source of confusion is Mark's joke about having traveled in George's plane twice, as changing Mark's dialogue from talking about a "first and...last" trip to a single trip results in a correct model prediction.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
MICE edits can also be used to debug model capabilities by offering hypotheses about "bugs" present in models: For instance, the edit in Table  3 might prompt a developer to investigate whether this PREDICTOR lacks non-literal language understanding capabilities. In the next section, we show how insight from individual MICE edits can be used to uncover a bug in the form of a dataset-level artifact learned by a model. In Appendix D, we further analyze the debugging utility of MICE edits with a PREDICTOR designed to contain a bug.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Manual inspection of some edits for IMDB suggests that the IMDB PREDICTOR has learned to rely heavily on numerical ratings. For instance, in the IMDB example in Table 3, the MICE edit results in a neg-  ative prediction from the PREDICTOR even though the edited text is overwhelmingly positive. We test this hypothesis by investigating whether numerical tokens are more likely to be edited by MICE.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We analyze the edits produced by MICE (GOLD + GRAD) described in §3.1. We limit our analysis to a subset of the 5K instances for which the edit produced by MICE has a minimality value of ≤0.05, as we are interested in finding simple artifacts driving the predictions of the IMDB PREDIC-TOR; this subset has 902 instances. We compute three metrics for each unique token, i.e., type t:
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
and report the tokens with the highest values for the ratios p r (t)/p(t) and p i (t)/p(t). Intuitively, these tokens are removed/inserted at a higher rate than expected given the frequency with which they appear in the original IMDB inputs. We exclude tokens that occur <10 times from our analysis.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Results from this analysis are shown in Table  4. In line with our hypothesis, we observe a bias towards removing low numerical ratings and inserting high ratings when the contrast prediction y c is positive, and vice versa when y c is negative. In other words, in the presence of a numerical score, the PREDICTOR may ignore the content of the review and base its prediction solely on the score (as in the IMDB example in Table 3).
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In this section, we reflect on MICE's shortcomings. Foremost, MICE is computationally expensive. Stage 1 requires fine-tuning a large pretrained generation model as the EDITOR. More significantly, Stage 2 requires multiple rounds of forward and backward passes to find a minimal edit: Each edit round in Stage 2 requires b × s × m decoded sequences with the EDITOR, as well as b×s×m forward passes and b backward passes with the PRE-DICTOR (with b = 1 the first edit round), where b is the beam width, s is the number of search levels in binary search over the masking percentages, and m is the number of generations sampled for each masking percentage. Our experiments required 180 forward passes, 180 decoded sequences, and 3 backward passes for edit rounds after the first.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
While efficient search for targeted edits is an open challenge in other fields of machine learning (Russell, 2019;Dandl et al., 2020), this problem is even more challenging for language data, as the space of possible perturbations is much larger than for tabular data. An important future direction is to develop more efficient methods of finding edits.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
This shortcoming prevents us from finding edits that are minimal in a precise sense. In particular, we may be interested in a constrained notion of minimality that defines an edit e(x) as minimal if there exists no subset of e(x) that results in the contrast prediction. Future work might consider creating methods to produce edits with this property.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We argue that contrastive edits, which change the output of a PREDICTOR to a given contrast prediction, are effective explanations of neural NLP models. We propose MINIMAL CONTRASTIVE EDITING (MICE), a method for generating such edits. We introduce evaluation criteria for contrastive edits that are motivated by human contrastive explanations-minimality and fluencyand show that MICE edits for the IMDB, NEWS-GROUPS, and RACE datasets are contrastive, fluent, and minimal. Through qualitative analysis of MICE edits, we show that they have utility for robust and reliable NLP system development.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
MICE is intended to aid the interpretation of NLP models. As a model-agnostic explanation method, it has the potential to impact NLP system development across a wide range of models and tasks. In particular, MICE edits can benefit NLP model developers in facilitating debugging and exposing dataset artifacts, as discussed in §4. As a consequence, they can also benefit downstream users of NLP models by facilitating access to less biased and more robust systems.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
While the focus of our work is on interpreting NLP models, there are potential misuses of MICE that involve other applications. Firstly, malicious actors might employ MICE to generate adversarial examples; for instance, they may aim to generate hate speech that is minimally edited such that it fools a toxic language classifier. Secondly, naively applying MICE for data augmentation could plausibly lead to less robust and more biased models: Because MICE edits are intended to expose issues in models, straightforwardly using them as additional training examples could reinforce existing artifacts and biases present in data. To mitigate this risk, we encourage researchers exploring data augmentation to carefully think about how to select and label edited instances.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We also encourage researchers to develop more efficient methods of generating minimal contrastive edits. As discussed in §5, a limitation of MICE is its computational demand. Therefore, we recommend that future work focus on creating methods that require less compute. 
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
For all datasets, f is initialized as a ROBERTA-LARGE model with a linear layer and maximum sequence length of 512 tokens. We train with AllenNLP (Gardner et al., 2017). For IMDB and NEWSGROUPS, we fine-tune f for 5 epochs with batch size 8 using Adam with initial learning rate of 2e−05, weight decay 0.1, and slanted triangular learning rate scheduler with cut frac 0.06. For RACE, we fine-tune f for 3 epochs with batch size 4 and 16 gradient accumulation steps using Adam with learning rate 1e−05, = 1e−08, and linear learning rate scheduler with 100 warm-up steps, and we fix f after the epoch with the lowest validation loss.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We use the transformers implementation (Wolf et al., 2020) of the base T5 for our EDI-TORS. We use Adam with a learning rate of 1e−4.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
For IMDB EDITORS, we use batch size 4 for all variants. For NEWSGROUPS, we use batch size 4 for fine-tuning with predictor labels and batch size 8 for fine-tuning with gold labels. For RACE, we use batch size 4 for fine-tuning with predictor labels and batch size 6 for fine-tuning with gold labels.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We remove newline and tab tokens (<br />, \t, \n) in all datasets, as these are tokenized differently by our PREDICTORS (ROBERTA-LARGE) and EDI-TORS (T5). For NEWSGROUPS, we also remove headers, footers, and quotes.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Inputs to EDITORS For IMDB and NEWS-GROUPS EDITORS, we simply prepend target labels to the masked original inputs. For RACE, we give the question, context, all answer options, and the correct choice as input to the RACE EDITOR. We only mask the context. See Table 5 for examples.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We noticed that generations sometimes degenerate when we decode from T5 with a large masking percentage n 2 . For example, sentinel tokens are sometimes generated out of consecutive order. We attribute this to the large difference between masking percentages we use (up to 55%) and masking percentage used during T5 pretraining (15%).
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Specifically, we observed that generations tend to degenerate after the the 28th sentinel token. Thus, we heuristically reduce the number of sentinel tokens by combining neighboring sentinel tokens that are separated by 1-2 tokens into one sentinel token. When the output degenerates, we do the following: In-fill the mask tokens with the "good" parts of the generation (i.e. parts with correctly ordered sentinel tokens), and replace the remaining mask tokens with the original text; get the contrast label probabilities from f for these intermediate in-filled candidates; of these, take the m = 3 candidates with the highest probabilities and use as input to generate m/m new candidates. 16 D Using MICE Edits to Debug a "Buggy" PREDICTOR: A Case Study
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
In §4, we illustrate how MICE edits can be used to debug both individual predictions and natural dataset artifacts learned by a model. Here, we further explore the utility of MICE edits in debugging through Data Staining (Sippy et al., 2020): We design a "buggy" PREDICTOR and evaluate whether MICE edits can recover the bug. We create a buggy RACE PREDICTOR by introducing an artifact into the RACE train set. This artifact is the presence of the phrase "It is interesting to note that" in front of the correct answer choice. We introduce this artifact as follows: We filter the RACE train data to contain instances for which the correct answer choice is contained by some sentence 17 and the overlapping sentence does not have a higher degree of n-gram overlap with some other (incorrect) choice. After filtering, 11,188 of 87,866 train instances remain. We then prepend "It is interesting to note that" to the overlapping sentence to design a correlation between the location of this phrase and the correct answer choice; our goal is to encourage a PREDICTOR to learn to predict the multiple choice option closest to this buggy phrase as the correct answer. If there are multiple overlapping sentences, we choose the one with the most overlap with the answer choice. We randomly sample from this filtered subset such that 10% of the train data contains this artifact. Our buggy RACE PREDICTOR is trained on this modified data using the same set-up from §A.1, except that we use a batch size of 2 and 32 gradient accumulation steps. 16 If one of the partially-infilled candidates results in the contrast label, we return this as the edited input. 17 A sentence "contains" the correct answer choice if the answer has at least a 4-gram overlap with the sentence.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Original Input Input to EDITOR NEWS Michael, you sent your inquiry to the bmw mailing list, but the sw replaces your return addr with the list addr so I can't reply or manually add you. please see my post re the list or contact me directly.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
label: misc. input: <extra_id_0>, you sent your <ex-tra_id_1> to the <extra_id_2>, but the <extra_id_3> your return <extra_id_4> with the list <extra_id_5> so I can't <extra_id_6> or <extra_id_7> add you. please see my post re the list or contact me directly.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
article: The best way of learning a language is by using it. The best way of learning English is using English as much as possible. Sometimes you will get your words mixed up and people wont understand. Sometimes people will say things too quickly and you cant understand them. But if you keep your sense of humor( ),you can always have a good laugh at the mistakes you make. Dont be unhappy if the people seem to laugh at your mistakes. Itś much better for people to laugh at your mistake than to be angry because they dont know what you are saying. The most important rule for learning English is "Dont be afraid of making mistakes. Everyone makes mistakes." question: In learning English, you should _. choices: speak as quickly as possible., laugh as much as you can., use it as often as you can., write more than you read.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
question: In learning English, you should _. answer: choice1: laugh as much as you can. context: The <extra_id_0> <extra_id_1>. Sometimes you will get your words <extra_id_2> <extra_id_3> <extra_id_4> have a good laugh at the mistakes you make. Don't be unhappy if the people seem to laugh at your mistakes. It's much better for people to laugh at your mistake than to be angry because they don't know what you are saying. The most important rule for learning English is "Don't be afraid of making mistakes. Everyone makes <extra_id_5>." choice0: speak as quickly as possible. choice1: laugh as much as you can. choice2: use it as often as you can. choice3: write more than you read. Original pred yp = (d) All
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Contrast pred yc = (b) Three
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Just as "Tiger Mom" leaves, here comes the "Wolf Daddy" called Xiao Baiyou. He believes he's the best parent in the world. Some days ago, Xiao Baiyou's latest book about how to be a successful parent came out. He is pretty strict with his four children. Sometimes he even beat them. But the children don't hate their daddy at all. And all of them finally went to Pecking University, It is interesting to note that three of them got good marks at Pecking University. And one of the top universities in China them even passed the exam without any problem. So Xiao proudly tells others about his education idea that children need strict rules. In his microblog, he said, "Come on, want your children to enter Peking University without rules? You must be joking." And, "Leave your children more money, and strict rules at the same time."But the "Wolf Daddy" way was soon questioned by other parents. Some say that Xiao Baiyou just want to be famous by doing so. The "Wolf Daddy" Xiao Baiyou is a 47-year-old Guangdong businessman who deals in luxury goods in Hong Kong. Unlike many other parents who usually have one child, Xiao has four children. Two of them were born in Hong Kong and two in the US. Some people on the Internet think the reason why his children were able to enter Peking University is because the exam is much easier taken from Hong Kong. The test accuracies of our original and buggy RACE PREDICTORS are both 84%, and so we cannot use this measure to select the better classifier. We ask whether MICE edits can be used for this purpose. One such edit is shown in Table 6. We observe that the signal from the edit, which contains both the manual artifact "It is interesting to note that" and the contrast prediction "three," is enough to overpower the signal from the explicit assertion that "All" is the correct answer ("And all of them finally went to Pecking University") such that the PREDICTOR's prediction changes to "Three." This edit thus provides evidence that some heuristic may have been learned by the predictor. Considering multiple MICE edits can validate such a hypothesis: We find that 17.2% of the edits produced by MICE reflect this bug (i.e. contain the phrase "interesting to note that"); in other words, they do uncover the manually inserted bug. Furthermore, MICE edits are able to uncover the artifact because they can insert new text. For instance, in the edit in Table 5, the buggy phrase "It is interesting to note that" is not part of the original input. Applying saliency-based explanation methods, such as gradient attribution, to the buggy PREDICTOR's prediction would not reveal the PRE-DICTOR's reliance on the manual artifact, as the buggy phrase is not already present in the text. This difference highlights a key advantage of MICE over existing instance-based explanation methods that attribute feature importance, which can only cite text already present in original inputs.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Contrast pred yc = positive With a catchy title like the Butcher of Plainfield this Ed Gein variation and Kane Hodder playing him will no doubt fly off the shelves for a couple of weeks.Most viewers will be bored laughed silly with this latest take on the life of Ed Gien. The movie focuses on Ed's rampage and gives us a(few)glimpses into his Psycosis and dwelling in Plainfeild.Its these scenes that give the movie a much needed jolt. What ruins this Another annoyance is the constant focus on other characters lives and focuses less on Eds.Big mistake here. Kane Hodder is a strange choice to play Gein,but He does pull it off quite well,and deserves more acting credits than he gets these days.Prascilla Barnes and Micahel Barryman also show up. 3/10 9/10
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Original pred yp = positive Contrast pred yc = negative I have just sat through this film again and can only wonder if we will see the likes kind of films like this anymore? The timeless music sex, the tender voices performances of William Holden and Jennifer Jones leave this grown man weeping suffering through joyous, romantic torturous, incoherent scenes and I'm not one who cries very often in life. Where have our William Holden's gone and will they make these moving, wonderful cynical, movies any more? It's sad to have to realize that they probably won't but don't think about it, just try to block that out of your mind. Even so Then again, they won't have Holden Shakespeare in it and he won't appear on that hill soap opera just once more either. You can only enjoy safely skip this film and watch it again.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Original pred yp = positive Contrast pred yc = negative This little flick is reminiscent of several other movies, but manages to keep its own style & mood. "Troll Trusty" & "Don't Be Afraid of the Dark" come to mind. The suspense builders performances were good, & just cross the line from G silly to PG uninteresting. I especially liked the non-cliche cliched choices with the parents; in other movies, I could predict the dialog ending verbatim, but the writing in this movie made better selections. If you want a movie that's not gross terribly creepy but gives you some chills, this is a great choice. 
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Original pred yp = talk Contrast pred yc = sci Would someone be kind enought to document the exact nature of the evidence against the BD NRA's without reference to hearsay or newsreports. I would also like to know more about their past record etc. but again based on solid not media reports. My reason for asking for such evidence is that last night on Larry King Live a so-called "cult space-expert" was interviewed from Australia who claimed that it was his evidence which led to the original raid discovery. This admission, if true, raises the nasty possibility that the Government acted in good faith, which I believe they did, on faulty evidence. It also raises the possibility that other self proclaimed cult space experts were advising them and giving ver poor advice.  until 1972 1973). The idea was to get the Giants Atheists to move into Shea Metlife Stadium. When a deal was worked out between the Giants Atheists and the Yankees Mets, the new AFL American franchise, the New York Titans Atheists, approached the City Mets about using the new stadium. The Titans Mets were playing in Downing Carling Stadium (where the Cosmos Atheists played soccer back in the 70s). Because Shea Stadium was tied into the World's Fair anyway, the city thought it would be a novel idea to promote the new franchise and the World's Fair (like they were doing with the Mets). So the deal was worked out. I'm under the impression that when Murph says it, he means it! As a regular goer to Shea, it is not a bad place since they've cleaned and renovated the place. Remember, this is its 30th Year!  The Internet has led to a huge increase in credit-card fraud. Your card information could even be for sale in an illegal web site. Web sites offering cheap goods and services should be regarded with care. On-line shoppers who enter can get credit-card information with stolen details through their credit-card information may never receive the online shopping sites, including buying goods they thought they bought. The thieves then go may use the information they have on your credit card to send shopping promotions, ads, or other Web sites. The thieves will not use with your card number -or sell the information over the Internet. Computers Recent developments in internet hackers have broken down security systems, raising questions about the safety of cardholder information. Several months ago, 25, 000 customers of CD Universe, an on-line music retailer, were not lucky. Their names, addresses and credit-card numbers were posted on a Web site after the retailer refused to pay US $157, 828 to get back the information. Credit-card firms are now fighting against on-line fraud. Mastercard is working on plans for Web -only credit card, with a lower credit limit. The card could be used only for shopping on-line purchases. However, But there are a few simple steps you can take to keep from being cheated. Ask about your credit-card firm's on-line rules: Under British law, cardholders have to pay the first US $ 7820 penalty of any fraudulent spending. And shop only activity at secure sites; Send your credit-card information only if the Web site offers advanced secure system. If the security is in place, a letter will appear in the bottom right-hand corner of your screen. The Website address may also start https: //-// // // andthe extra "s" stands for secure. If in doubt, Never give your credit-card information over the telephone. Keep your password safe: Most on-line sites require a user name and password before when placing an order. Treat your passwords with care.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Question: If you want to be a football player, you should __.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
(a) buy a good football (b) play football (c) watch others play football (d) put your football away Original pred yp = (b)
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Contrast pred yc = (a)
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
We are all learning English, but how can we learn English well? A student can know a lot about English, but maybe he can't speak English. If you want to know how to swim be a football player, you must get into the river buy a good football. If And if you want to be a football an English player, you must play football. So, you see. You can learn English only by using it. You must listen to your teacher in class. You must read your lessons every day. You must speak English to your classmates and also you must write something sometimes. Then one day, you may find your English very good.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Question: This story most probably took place __.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
(a) at the beginning of the term (b) in the middle of the term (c) at the end of the term (d) at the beginning of the school year Original pred yp = (c) Contrast pred yc = (b)
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
A teacher stood was giving new classes to students in front the middle of his history this term. The students were in class of twenty students just before handing out the final exam. His students by now. They sat quietly and waited for him to speak. "It's been a pleasure teaching you this term my last chance," he said to them. The class started to cry. They cried for a long time. Finally, the teacher got up. He looked them in surprise. Then he asked them to leave. They "You've all worked very hard, so I have a pleasant surprise for you. Everyone who chooses not to take the final exam will get a 'B' for the course." Most of the students jumped out of their seats. They thanked the teacher happily, and walked out of the classroom. Only a few students stayed. The teacher looked at them. "This is your last chance," he said. "Does anyone else want to leave?" All the students there stayed in their seats and took out their pencils. The teacher smiled. "Congratulations," he said. "I'm glad to see you believe in yourselves. You all get A on well." Table 9: Examples of edits produced by MICE for inputs from the RACE dataset. Insertions are bolded in red.
Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Deletions are struck through. y p is the PREDICTOR's original prediction, and y c the contrast prediction. True labels for original inputs are underlined.
Generating Realistic Natural Language Counterfactuals
The increase of machine learning (ML) applications in high-stakes domains has led to a proliferation of Explainable AI (XAI) and Interpretable ML approaches, aimed at making models (global explanations) or individual decisions (local explanations) more understandable (Doshi-Velez and Kim, 2017;Tomsett et al., 2018). Output explanations explain individual decisions by understanding the (local) behavior around the output (Guidotti et al., 2019). However, in practice individuals may not always have access to the models they want explained (e.g. because of intellectual property) (Edwards and Veale, 2017). To overcome this access problem, model-agnostic approaches (sometimes called post-hoc approaches (Lipton, 2016)) only require access to the model outputs for provided instances, with the added benefit of being applicable to explain any model for a type of ML task (Ribeiro et al., 2016a). Prominent model-agnostic output explanations are local surrogate models (Ribeiro et al., 2016b), feature importances (Lundberg and Lee, 2017;Fong and Vedaldi, 2017), example-based ex- planations (Kim et al., 2016) and counterfactual explanations (Wachter et al., 2018).
Generating Realistic Natural Language Counterfactuals
Counterfactual explanations express what might have happened instead (Roese and Olson, 1995): certain values in an input instance are perturbed (e.g. the age of a defendant) while keeping other values the same, in order to observe how that influences the output (e.g. they would not have been convicted). Each of the output-changing perturbations is a counterfactual, where the difference between the counterfactual and the original instance provides insights into how the inputs affect the outputs, and can be used to pinpoint fairness issues and to reach a desired output (Wachter et al., 2018). As these counterfactuals are a valuable means to understand the behavior of a system, in recent years the same technique has been applied for explaining ML decisions-mainly for structured data (e.g. (Russell, 2019;Ustun et al., 2019;Wachter et al., 2018)) and image data (e.g. (Dhurandhar et al., 2018;Guidotti et al., 2019;Poyiadzi et al., 2020)).
Generating Realistic Natural Language Counterfactuals
Unlike structured and image data, counterfactuals for natural language text data have largely been disregarded. For text classification, Martens and Provost (2014) proposed the removal of words as a means to measure their contribution to the output. This paradigm was later adopted for constructing model-agnostic local surrogate models (Ribeiro et al., 2016a) and to determine which words have to be necessarily present for a classification decision (Ribeiro et al., 2018). Yet, this paradigm fails to create realistic counterfactuals (as illustrated in Figure 1 for sentiment analysis). Realisticness is an important property for how humans create and accept counterfactuals (Byrne, 2019;Miller, 2018) and may help prevent misleading explanations produced by model-agnostic explanation methods (Slack et al., 2020). Recently, human-inthe-loop approaches were proposed (Ribeiro et al., 2020;Wu et al., 2021) to support explainees in forming realistic counterfactuals.
Generating Realistic Natural Language Counterfactuals
In this work, we propose CounterfactualGAN: a method able to model-agnostically generate realistic, targeted counterfactuals for natural language text regression and classification without explainee intervention. Our method (i) generates realistic counterfactuals for the text-domain-ensuring dataset-specific realisticness by adversarially training on a training set-, (ii) uses a single model to provide counterfactuals for any instance and any (classification or regression) target of a black-box model, (iii) generates counterfactuals with a single pass after training, and (iv) does not require explainee intervention to do so.
Generating Realistic Natural Language Counterfactuals
In the literature, several properties of counterfactual generation methods for ML classification and/or regression models have been suggested. First, the generation of counterfactuals is either targeted (i.e. to a specific class or regression output) or untargeted (any target other than the original) (Zhang et al., 2020). Second, while generally viewed as being part of the post-hoc XAI approaches, some methods assume white-box access to the model rather than viewing it as a black-box (e.g. (Russell, 2019;Ustun et al., 2019)). For example, if the model is a linear classifier and its weights are accessible, these weights can be used to effectively find (targeted) counterfactuals. Third, for each original instance one or multiple counterfactuals can be found, providing either a single explanation or elucidating the various ways in which decisions may change (Wachter et al., 2018). When selecting multiple counterfactuals, approaches are typically concerned with the diversity of the counterfactuals to ensure maximal coverage with a sparse counterfactual set (Russell, 2019;Karimi et al., 2020).
Generating Realistic Natural Language Counterfactuals
While these properties impact the approach for obtaining counterfactuals, further strategies are required to confine the search space of possible counterfactuals to the ones that hold the best explanatory value. Some approaches select the nearest counterfactuals (e.g. (Wachter et al., 2018)), such that minimal changes are required to change to the counterfactual. However, more recently authors have addressed the issue of implausibility: "[...] the counterfactuals generated may not be valid datapoints in the domain or they may suggest featurechanges that are difficult-to-impossible" (Keane and Smyth, 2020, pp. 166-167). Implausibility has been tackled with various strategies: either enforcing user-imposed feasibility constraints (e.g. excluding explanations where one needs to lower their age to lower the risk of recidivism) (Poyiadzi et al., 2020;Karimi et al., 2020) or using automated methods, such as selection based on closeness to a class prototype (Van Looveren and Klaise, 2021), that on the path from changing from the factual to counterfactual no other outputs are encountered (Laugel et al., 2019) or selecting instances from the training set as counterfactuals instead of generating them (Keane and Smyth, 2020).
Generating Realistic Natural Language Counterfactuals
Many counterfactual generation methods mainly consider structured and image data. Two methods that do support the creation of counterfactuals for natural language text require humans to determine where to apply changes in the instance: CheckList (Ribeiro et al., 2020) suggests input perturbations to a user, who can then choose from these perturbations and test how they affect the output, while PolyJuice (Wu et al., 2021) uses control codes in a finetuned GPT-2 model to form counterfactuals. MiCE (Ross et al., 2021) can generate counterfactuals using a finetuned T5 model with white-box access to a predictive model. However, optimizing textual perturbations to find counterfactuals with black-box access in a fully automated manner poses specific problems, as encountered in the related areas of adversarial ML (seeking semantically imperceptible changes to text that change the black-box label) and style transfer (changing linguistic attributes of groundtruth texts, while retaining content). First, it is nontrivial how to define distance measures between the original instance and its perturbations, as they typically are discrete objects (Belinkov and Glass, 2019). Second, minimizing this distance cannot easily be formulated as an optimization problem, as this requires computing gradients on discrete inputs (Belinkov and Glass, 2019). The adversarial attack and style transfer literature tackles these problems by either (i) leveraging a combination of NLP algorithms and knowledge-engineered perturbation rules (e.g. (Li et al., 2019;Jin et al., 2020)), or (ii) finding perturbations in a latent space of an autoencoder model and decoding these into natural language instances (Melnyk et al., 2017;.
Generating Realistic Natural Language Counterfactuals
We draw from insights in these areas, especially adversarial attacks-able to craft instances changing a black-box prediction-, but observe that their goal of imperceptibility of the perturbation (Zhang et al., 2020) (i.e. humans being unable to tell the difference between two instances and assigning the same label) is at odds with the goal of counterfactuals used in explanations to be humanunderstandable (Wachter et al., 2018). Counterfactuals provide explainees (e.g. developers, layusers or examiners (Tomsett et al., 2018)) with meaningful information regarding a models' prediction, without necessitating technical know-how (Wachter et al., 2018). This requires an explainee to perceive the difference between the original instance and counterfactual (e.g. which words have changed), potentially even with limited technical or domain expertise. We note this perceptibility does not have to align with human judgments of distinguishing factors in a task: changing from 'him' to 'her' in sentiment analysis may be equally perceptible as changing from 'good' to 'bad'.
Generating Realistic Natural Language Counterfactuals
For natural language text, we propose a new strategy to create plausible counterfactuals in an automated manner, by applying a realisticness constraint to the generated counterfactuals. This approach has several benefits over the strategies tackling implausibility in Section 2.1: (i) no domainspecific assumptions have to be made by users, (ii) the perturbation path between an original instance and its counterfactual is not constrained and (iii) counterfactuals are not restricted to only training instances. What realisticness entails may depend on the type of language expected within a certain context, e.g. realistic movie reviews typically differ in use of language and grammatical correctness from Tweets. Therefore, we deem an instance realistic if it is indistinguishable from other instances (expected) in a dataset from a specific domain.
Generating Realistic Natural Language Counterfactuals
Definitions. Let us assume we are able to provide inputs to a black-box ML model f : X → Y and get the corresponding predictions, and have an instance x ∈ X for which to find counterfactuals. f (·) was trained on a dataset containing N labeled or unlabeled instances (represented by respectively). Instance x is represented as a feature vector x = (x 1 , x 2 , ... , x n ), where each x denotes a feature value. In our work, x i is either a word-level token or is absent. A second index j (x i,j ) is used sometimes when tokens are represented by a probability distribution over a vocabulary of M tokens (including a special token indicating absence).
Generating Realistic Natural Language Counterfactuals
A perturbed instance x = x + δ is formed by applying one or more valid perturbations δ to x (Dhurandhar et al., 2018). A valid perturbation transforms the feature values in x such that x ∈ X . For example, valid perturbations are word replacements or removing a word by setting it to absent. We estimate instance realisticness by determining if it is indistinguishable from the original data distribution p(x). In practice we estimate this indistinguishability with a discriminator model g X : X → [0, 1] (trained on X) indicating the likelihood that an instance x could have come from p(x). A low score indicates out-of-distribution instances.
Generating Realistic Natural Language Counterfactuals
We define the set of counterfactuals (CFs) of the instance Karimi et al., 2020), i.e. all instances with an output different to x. Targeted counterfactuals (TCFs) are instances that change the fact (the current output y = f (x)), to the foil, an output of interest y ∈ Y. Instead of requiring the output f ( x) of perturbed instance x to be exactly equal to y , we generalize the assumption f ( x) = y and obtain:
Generating Realistic Natural Language Counterfactuals
is to output of interest y and ≥ 0 a user-defined threshold for the strictness in including instances. 1 A realistic (targeted) counterfactual is a (targeted) counterfactual that also maximizes realisticness model g X (·).
Generating Realistic Natural Language Counterfactuals
We operationalize model-agnostic, realistic, targeted counterfactuals for text regression and classification with our proposed method Counterfactual-GAN. Our method works across predictive models by crafting counterfactuals in the form of a string. To ensure counterfactual realisticness, we combine insights from pretrained language models (LMs) and generative adversarial networks (GANs).
Generating Realistic Natural Language Counterfactuals
GANs use a generator G and discriminator D to learn latent representations or mappings in an unsupervised or semi-supervised manner (Creswell et al., 2018). By letting these networks compete, they are forced to jointly improve their performance. After training, generator G is used to generate realistic synthetic instances.
Generating Realistic Natural Language Counterfactuals
Rather than having G unconditionally generating realistic instances, Conditional GANs aim to create realistic mappings for specific inputs (Creswell et al., 2018). For example, an unconditional GAN may be able to create sentences with positive sentiment, while a Conditional GAN can turn the positive sentiment of an input sentence into a negative sentiment counterpart. First popularized in the image domain, approaches such as pix2pix  are able to e.g. convert grayscale photographs to full color and convert day scenes to night scenes. CycleGAN (Zhu et al., 2017) is able to do this without ground-truth pairs with example mappings between domains by jointly training two generators G ab and G ba -where G ab learns a mapping from domain a (e.g. positive sentiment) to b (e.g. negative sentiment), while G ba learns a mapping from domain b to a. An important contribution of CycleGAN is ensuring a minimal reconstruction loss of the network G ba (G ab (x)) ≈ x, helping the network to preserve relevant input features.
Generating Realistic Natural Language Counterfactuals
The downside of using CycleGAN is that in the multi-domain case (e.g. multi-class or a continuous domain) conditional generation requires training many separate generators and discriminators. Star-GAN (Choi et al., 2018) mitigates this shortcoming of CycleGAN by using one generator G that takes both the input instance and a target domain (e.g. positive/negative sentiment) as inputs, and a single discriminator D that predicts (1) if the instance is real [D adv ] and (2) which target domain it belongs to [D tgt ].
Generating Realistic Natural Language Counterfactuals
Language models (LMs). While GANs have shown promising results, their application to natural language text has been limited. The discrete nature of text makes propagating the gradient from the discriminator back to the generator infeasible. We therefore opt to use the approach of finding a mapping in latent (embedding) space Z (in similar vein to e.g. (Melnyk et al., 2017)), but in our case use a pretrained LM for the autoencoder. Pretrained LMs have proven to greatly improve the state-ofthe-art performance on a plethora of down-stream tasks (e.g semantic similarity, reading comprehension and commonsense reasoning) (Radford and Salimans, 2018). By using a pretrained LM, we leverage its adeptness in encoding syntax and semantic content-even beyond the training data.
Generating Realistic Natural Language Counterfactuals
CounterfactualGAN. Our method combines the encoder-decoder architecture of an LM and the generator-discriminator architecture of a GAN for finding counterfactuals. Discriminator D is responsible for determining the realisticness of an instance x, while we use the predictions of black-box model f (·) to determine if a counterfactual is of output y .
Generating Realistic Natural Language Counterfactuals
In practice, we use BERT (Devlin et al., 2019) as an LM encoder to create an embedding z. Decoder Dec(·) is then tasked with mapping z back to original instance x. For the GAN, we use a StarGAN (Choi et al., 2018), with a single generator G-that is provided with a target y -, and one discriminator with two heads, tasked with determining whether the instance was real or fake (D adv ) and how well the instance corresponds to the target (D tgt ). To ensure that the output is similar after mapping to the target domain and back, the reconstruction loss is not only calculated on the embeddings z and z = G(G(z, y ), y), but also on x and the token predictions according to the decoder x = Dec(z ).
Generating Realistic Natural Language Counterfactuals
As our goal is to provide counterfactual explanations, unlike the original StarGAN, D tgt is trained on the predicted labels y = f (x) of the black-box decision function we aim to explain rather than ground-truth labels. Because black-box f (·) uses instances x ∈ X to make its predictions rather than embedding z ∈ Z, D tgt is first trained to distinguish target outcomes using embedding z. In addition, as our method relies on a highly accurate mapping of the encoder-decoder part of the model, the encoder and decoder are pretrained on the training data as well. To incorporate these two requirements, we propose to train Counterfactual-  GAN in two phases, both illustrated in Figure 2.
Generating Realistic Natural Language Counterfactuals
Phase 1 (Figure 2a) starts with a pretrained LM encoder and decoder, and has the goal of (i) ensuring that the decoder accurately reconstructs embedding z into the encoded instance x, and (ii) ensuring that D tgt accurately mimics black-box f (·). Next, Phase 2 (Figure 2b) fixes the encoder and decoder weights, and introduces generator G to find the mapping in embedding space Z. Generator G first maps z to z with target y , and then back to original outcome y-resulting in z . For the discriminator, z is marked as a real instance of target y, z as a fake instance of target y and z as a fake instance of original target y. Counterfac-tualGAN uses a three-layer Transformer decoder (Vaswani et al., 2017) for generator G, while discriminator D uses a two-layer GRU to combine embedding z into a single low-dimensional embedding for the entire input, used by the two heads D adv and D tgt . After both phases, a counterfactual is generated from x with target y by running its encoding through generator G and decoding the generated embedding: x = Dec(G(Enc(x), y )). During this generation, a top-k of counterfactuals is generated for each instance, from which one string (i.e. the counterfactual) that is most similar to target y according to f (·) is returned to the end-user.
Generating Realistic Natural Language Counterfactuals
Training objectives. To generate instances that are indistinguishable from real instances, discriminator D uses an adversarial loss
Generating Realistic Natural Language Counterfactuals
where z is a (generated) embedding and r a value indicating whether the instance was real (1) or fake (0). G tries to minimize this objective, while D tries to maximize it. Next to the adversarial loss we also include a target loss L tgt to ensure that the generated instances (embedded as v = z ) resemble the target domain, while the original instances (embedded as v = z) resemble the original domain (Choi et al., 2018). To handle multiple types of black-box methods, we further distinguish between classification and regression targets:
Generating Realistic Natural Language Counterfactuals
Here, w is either the original label y (in case of v = z) or the target label y (in case of v = z ) corresponding to that respective instance. We indicate the version used by the discriminator (with z and y) as L D tgt , while we indicate the version used by the generator (with generated embedding z = G(z, y ) and y as target label) as L G tgt . Both G and D try to minimize this objective.
Generating Realistic Natural Language Counterfactuals
Lastly, the reconstruction loss (Zhu et al., 2017) L rec = 1 2 L rec,x + 1 2 L rec,z ensures that only domain-relevant parts of the inputs are changed when constructing a counterfactual. Here, we use a cross-entropy loss L rec,x between original instance x and the cycle-reconstructed instance x = Dec(z ) and the L 2 -norm L rec,z of their respective embeddings z and z = G(G(Enc(x), y ), y):
Generating Realistic Natural Language Counterfactuals
where y is the original label, y the target label, and x i and x i (with the j-th token in a vocabulary) are corresponding elements of sequence x and x , respectively. G aims to minimize this objective.
Generating Realistic Natural Language Counterfactuals
In Phase 1 (finetuning) we train the encoderdecoder part with a language modelling loss L lm and pretrain discriminator D by jointly training D adv (L adv ) and D tgt (L D tgt ). To increase the informativeness of instances during finetuning, in some instances words are randomly swapped or commonly used tokens belonging to ground-truth instances with the approximate target values are inserted. The goal for Phase 1 is to minimize L f inetune using the aforementioned loss functions:
Generating Realistic Natural Language Counterfactuals
where L adv is the adversarial loss, L D tgt the discriminator target loss and L lm the language modelling loss. λ D tgt and λ lm are user-defined hyperparameters indicating the objectives' relative weights.
Generating Realistic Natural Language Counterfactuals
In Phase 2, the generator and discriminator are jointly optimized in an adversarial setting. They are trained using objective L GAN :
Generating Realistic Natural Language Counterfactuals
where G tries to minimize objective L GAN and D tries to maximize it. Generator loss L G comprises the adversarial loss L adv , the target loss L G tgt for the generator, and reconstruction loss L rec (responsible for ensuring minimal change when mapping to the target label and back). Again, λ G tgt and λ rec are user-defined hyperparameters. The implementation details for our method are included in Appendix A.
Generating Realistic Natural Language Counterfactuals
CounterfactualGAN was evaluated against four baselines using a quantitative validation and a human evaluation in the form of a user experiment.
Generating Realistic Natural Language Counterfactuals
We evaluated the generation methods using three task-specific datasets: one regression analysis task, one binary classification task and one multi-class classification task. To assess model-agnosticism, for each of these tasks three models were devised.
Generating Realistic Natural Language Counterfactuals
Datasets & tasks. We used three well-known NLP datasets for the training and evaluation of the predictive models and generation methods. These datasets cover various domains of NLP regression and classification tasks. HATESPEECH (Davidson et al., 2017) is a Twitter dataset used for hatespeech identification, where for our purposes the three class labels hatespeech (1.0), offensive language (.4) or neither (.0) were recoded to be used in a regression analysis of hatespeech severity. During preprocessing, @mentions in Tweets were anonymized with a string '@user'. The Stanford Sentiment Treebank (Socher et al., 2013) (SST-2) contains movie reviews with either positive or negative sentiment. SNLI (Bowman et al., 2015) is a textual entailment dataset, where the goal is to determine whether a hypothesis entails, contradicts or is neutral to a premise.
Generating Realistic Natural Language Counterfactuals
Each of these datasets was split into a training set (used for predictive/counterfactual generation model training), development set (used for hyperparameter optimization) and test set (used for evaluation). An overview of each dataset is provided in Table 1, including the task description, size of the dataset and its mean number of words. 2 Predictive models. The predictive methods result in a model f (·), which gives predictive values (regression values or class probabilities) for each instance. First, we include a hand-crafted whitebox model (WB) where ground-truth counterfactuals can be deduced. 3 In addition, we used two recent popular approaches that have shown competitive performance on several text regression and classification tasks as black-box models: InferSent (IS) (Conneau et al., 2017) and BERT (BE) (Devlin et al., 2019). Both models were finetuned on the specific dataset and corresponding task. The performance of each method on each tasks is shown in Table 1, where the performance is measured with MSE (lower is better) for HATESPEECH and macroaveraged F 1 (higher is better) for SST-2 and SNLI.
Generating Realistic Natural Language Counterfactuals
We compared CounterfactualGAN to four baseline model-agnostic counterfactual generation methods. Each method creates a single counterfactual x = TCF(x, y ) for each instance x ∈ X test .
Generating Realistic Natural Language Counterfactuals
Classification (Martens and Provost, 2014) aims to find the minimal set of words so that removing these words changes the decision from the current  (Miller, 1995) to craft untargeted adversarial examples for classification. We extend this method for targeted counterfactual generation by (i) also allowing for antonym substitutions, (ii) including support for regression analysis and (iii) returning an instance close to the target. TextFooler. TextFooler (Jin et al., 2020) provides a competitive baseline for semantic adversarial attacks for text classification and entailment. It replaces the most sensitive words with synonyms with an equal part-of-speech to craft adversarial instances, while ensuring maximal semantic similarity. For our purposes, we extend TextFooler to regression analysis by making two predictions y and y approximately equal when |y − y | ≤ 0.2.
Generating Realistic Natural Language Counterfactuals
A realistic, targeted counterfactual generation method should produce counterfactuals that (i) accurately mimic black-box f (·), (ii) are realistic for a given dataset (see Section 3) and (iii) are perceptibly distinguishable from the original instance (see Section 2.2). To capture these aspects, we evaluated the generation methods on each predictive model and task using three metrics: fidelity, naturalness and perceptibility. Fidelity determines how accurately the method captures f (·), high naturalness indicates realisticness and perceptibility quantitatively estimates if the difference between x and x is sufficient to be used for forming explanations. The quantitative metrics fidelity and perceptibility were evaluated on test set X test 4 , while we selected a representative subset of 30 instances in X test to evaluate naturalness in a human experiment. We assigned a random target y to each instance x ∈ X test , and used this to generate a corresponding targeted counterfactual x = TCF f (x, y ) using each generation method. This procedure was repeated for five random targets per instance, resulting in five counterfactuals for each instance. For CounterfactualGAN, the counterfactual for each instance was selected by generating the top-5 counterfactuals and selecting the one where f ( x) was closest to target y . Table 2 shows example targeted counterfactuals. Additional examples are included in Appendix B.
Generating Realistic Natural Language Counterfactuals
Fidelity. Fidelity evaluates how well the generation method estimates the true behavior of the black-box predictive model (Ribeiro et al., 2016b). A generation method accurately mimicking its black-box will be able to produce counterfactuals that are of target class y according to predictive model f (·). For classification, the fidelity is oftentimes captured using the label flip score (Wu et al., 2021), i.e. how often the predicted label 'flips' to the target label. To generalize this notion beyond classification, for each instance we compared output f ( x) to target y and measure its performance. The measures used are the same as for overall predictive model performance for each task. Table 3 reports on the results for the fidelity evaluation, showing that our method outperforms the baselines on 6 out of 9 model-dataset pairs. A one-way ANOVA shows a significant difference between methods for HATESPEECH [F (4, 70)  Masked-LM a blond young woman in a black blue shirt is standing being a counter seenly seated, in a large chair.
Generating Realistic Natural Language Counterfactuals
neutral TextFooler a blond blonds woman lady in a black negra shirt jumper is standing stands behind a counter counteract.
Generating Realistic Natural Language Counterfactuals
contradiction CounterfactualGAN a blond big woman man in a black shirt is standing behind a counter. contradiction Perceptibility. To be used in explanations, the generation methods should produce counterfactuals x that are perceptibly distinguishable from their corresponding instances x (see Section 2.2). We quantitatively estimate perceptibility by taking semantic similarity estimated by the Universal Sentence Encoder (USE) (Cer et al., 2018), where we measure perceptibility with semantic distance 1 − USE(x, x). Unlike semantic adversarial examples, which have the goal of minimizing this semantic distance, we aim to have a higher score such that the difference (e.g. positive words in a review to negative ones) can be easily perceived-while being far enough from a completely unrelated counterfactual (score of 1). Human experiment: naturalness. We qualitatively determined which of the generation methods produces the most natural counterfactuals according to 196 native English speakers sampled from crowdsourcing platform Prolific 5 . Naturalness in-5 https://prolific.co dicates how realistic an utterance is for a given context ('movie reviews ', 'Tweets' or 'reading comprehension'). Participants were provided with pairs of counterfactuals generated for the same instance and predictive model, and asked which utterance was more natural in that context. A natural instance is one that could have been produced by a human (Novikova et al., 2017). Note that unlike other experiments humans were not asked to judge if the counterfactual correctly belongs to the target (e.g. positive/negative reviews), as the counterfactual explains the model behavior on the data-which may not correspond with human interpretation of the distinguishing factors. Each participant received a random subset of 50 pairs in which they chose whether they prefer the first utterance (generated by one counterfactual generation method), the second (generated by another), or had no preference. The participants were urged to choose between the utterances even with a slight preference. All instances for each predictive model, dataset and generation method were shown at least five times to varying participants. Participants had excellent inter-rater reliability [Krippendorff's α = .84]. Appendix C expands further on the experimental procedure. The results, reported in Table 5, show that our method is preferred (wins) regarding naturalness across all datasets and predictive models.
Generating Realistic Natural Language Counterfactuals
In this paper, we proposed CounterfactualGAN: a counterfactual generation method providing realistic counterfactuals to explain natural language text regression and classification black-box models, using a combination of pretrained LMs and a StarGAN to craft counterfactuals. Experimental results showed that our counterfactual generation method outperforms baselines across predictive HATESPEECH (↓MSE)     Table 4: Mean perceptibility of each counterfactual generation method (5 run average). The best score (↑) for each column is highlighted in bold.  Table 5: Percentage (%) of wins (W), ties (T) and losses (L) of our method against four baseline methods according to human judgments of naturalness. The methods deemed most natural are highlighted in bold. models and datasets in finding human-perceptible, targeted counterfactuals, which remained natural according to human judgments.
Generating Realistic Natural Language Counterfactuals
CounterfactualGAN greatly improves natural language counterfactuals' quality, potentially having a profound effect on the explanation quality of model-agnostic XAI methods using perturbations to form explanations (e.g. local surrogates (Ribeiro et al., 2016b) and counterfactual explanations (Wachter et al., 2018)). For future work, we intend to (i) assess for which XAI methods and in which contexts realisticness is most beneficial, (ii) determine what level of perceptibility is optimal for human-understandable explanations, and (iii) extend CounterfactualGAN to other languages than English and more ML task types (e.g. multi-label classification).
Generating Realistic Natural Language Counterfactuals
CounterfactualGAN is implemented in PyTorch (Python 3.7.5) and trained on a Tesla V100 GPU with CUDA 10.2. BERT is used as a pretrained language model (LM) encoder Enc(·), implemented using prajjwal1/bert-small of the Transformers package (Wolf et al., 2020)-allowing more effective training of the method parts than a larger model. bert-small is a small, uncased version of the BERT model from the official Google repository 6 with a hidden dimension size of 512, 4 attention heads and 4 layers that total to 29.1M parameters (Turc et al., 2019).
Generating Realistic Natural Language Counterfactuals
The encoder transforms the instance into an embedding of size t × h, where t are the maximum number of tokens (where before the first token we place the special [CLS] token and [SEP] after the last token, and fill the remainder with [PAD] tokens) and h the hidden dimension size. Decoder Dec(·) is a fully-connected linear layer with bias, transforming an embedding z into a tensor t × v containing logits for each token in a vocabulary of size v. We extract k token sequences from these using nucleus sampling with p = .9 (Holtzman et al., 2020). Nucleus sampling selects the top logits for each t such that their softmax probabilities sum to p. The chosen tokens are then recombined into strings using the Penn Treebank detokenizer in NLTK 7 for each of the top-k counterfactuals. These were then fed back into f (·) to calculate their true target, after which the one most similar to the provided target y was selected as the counterfactual.
Generating Realistic Natural Language Counterfactuals
During Phase 1, we increase instance informativeness by including copies of a batch where (i) in each instance 15% of the words are replaced by a [MASK] token, (ii) in each instance 15% of the tokens are randomly swapped and (iii) for 15% of the tokens random tokens belonging to the target (regression value bin or class) are inserted into the token sequence. In each case, this is done non-destructively to ensure that the special tokens [CLS] and [SEP] are not replaced.
Generating Realistic Natural Language Counterfactuals
Generator G is a three-layer Transformer decoder with three attention heads, which receives the embedding Enc(x) as a shared input. The target tokens are the same embedding, except that the first token is replaced with an embedding of size h that contains the target for that instance. Discriminator D is a two-layer gated recurrent unit (GRU) with 10% dropout, which transforms an embedding t × h into an embedding 1 × h that is used by the D tgt and D adv heads. Both D adv (determining realisticness of instances) and D tgt (predicting what the value of black-box f (·) for the embedding is) are single layer feed-forward neural networks.
Generating Realistic Natural Language Counterfactuals
Hyperparameters. All networks are optimized with the Adam optimizer (Kingma and Ba, 2014) with a learning rate of .0002, β 1 = .9 and β 2 = .999. During Phase 2 (GAN training), for each time G performs one step, D performs two. We use hyperparameter optimization to find the relative weights λ of parts of the loss functions, the number of epochs in Phase 1 (finetune epochs) e 1 , and the number of epochs in Phase 2 e 2 . Optimization was performed for 30 trials for each modeldataset pair using random combinations formed by Python package optuna. 8 The goal of this optimization was maximal fidelity (minimal M SE for HATESPEECH, maximal macro-averaged F 1 for SST-2 and SNLI). The search ranges for the hyperparameters were as follows: (i) e 1 (Phase 1 epochs) in range [7, 10]; (ii) e 2 (Phase 1 epochs) in range [30,100]; (iii) λ lm a choice of value from (1,3,5,10,15,25,50,100), and; (iv) λ D tgt,1 (during Phase 1), λ D tgt,2 (during Phase 2), λ G tgt and λ rec a choice of value from (1,3,5,10,15,25,50).
Generating Realistic Natural Language Counterfactuals
For reproducibility purposes, the hyperparameters of the model with the highest fidelity for each predictive model-dataset are reported in Table 6  Time usage. The training and inference time vary by dataset due to their different sizes (see Table 1). For the hardware setup that was previously reported, PyTorch on a single core of the Tesla V100 GPU, we report the mean wall time for train-ing and the wall time for crafting counterfactuals for the whole test set (inference).  Predictive models. InferSent (Conneau et al., 2017) (IS) uses a bi-directional LSTM on wordlevel GloVe embeddings (Pennington et al., 2014) to create semantic representations of sentences. In BE the [CLS] token in the final layer is used as a sentence representation. In both cases, these were then input into a linear layer to produce the prediction for the black-box predictive model. Table 8 compares illustrative examples from all datasets for all counterfactual generation methods. In addition, we first include the original instance, and described its original prediction and the counterfactual target. Note that all instances and generated counterfactuals have been lowercased. Moreover, in HATESPEECH users in mentions are replaced by '@user' to ensure anonymity.
Generating Realistic Natural Language Counterfactuals
For our human experiment, we recruited 199 participants from crowd-sourcing platform Prolific. All users remained anonymous. Their Prolific ID was only used for participant pay-out (they were awarded £2.50 for 20 minutes of their time, providing a good hourly rate according to https: //prolific.co in January 2021), after which it was discarded in further processing. Participants were randomly assigned to us, where the selection criteria we provided was that their first spoken language is English (self-reported) and had an approval rate of at least 80%. First, participants were introduced to the task to determine the naturalness of two utterances. We defined naturalness in our study as " [...] an utterance is more natural if it is more likely that it was produced by a human. Aspects you could consider are the type of language used in a context, grammatical correctness and semantically meaningful sentences." Next, they were asked to agree to a GDPR-compliant informed consent form before continuing their participation. We generated pairwise comparisons by sampling 30 instances from each test set, and for each explanation method picking the corresponding counterfactuals from the run with the best fidelity score (highest F 1 or lowest M SE). Participants were provided with 50 pairwise comparisons, where for each question they were asked "Which of the following {context} is more natural?" The provided text in the {context} placeholder depended on the dataset these counterfactuals were generated for, namely Tweets for HATESPEECH, movie reviews for SST-2 and reading comprehension sentences for SNLI. Figure 3 provides an example question. The 50 questions were randomly drawn from all pairwise comparisons, and shown in a random order. To check the quality of each submission, we included two quality control mechanisms: (i) we recorded the time of each survey completion and (ii) we included two control pairwise comparisons before and after the pairwise comparisons. The estimated completion time of the survey was 20 minutes, with a true average completion time of 14 minutes and 42 seconds. The options in the control questions compared true instances to ones with the lowest word-level edit distance by any generation method, one for HATESPEECH and one for  Excluding participants with completion times ≤ 5 minutes (n = 2) and participants choosing the nonnatural answer for both control questions (n = 1), the final sample size was n = 196.
Generating Realistic Natural Language Counterfactuals
Method Original instance and generated counterfactuals HATESPEECH (black-box IS, higher to lower hatespeech score)
Generating Realistic Natural Language Counterfactuals
Original rt @user: i aint gonna text first cus pride b*tch SEDC @user i aint cus PWWS+ rt user: i gonna text first cus humility Masked-LM rt @user: i destroyedt gonna costumes & natural turner promising b*tch bree TextFooler rt @user: i aint gonna text first cus pride bitch Ours (top-1) rt @user: i aint na text first cus pride Ours (top-3) rt @user: i aint gonna text first cus my pride Ours (top-5) rt @user: i aint gonna text first cus my pride HATESPEECH (black-box WB, medium to higher hatespeech score)
Generating Realistic Natural Language Counterfactuals
Original rt @user: sonia never criticises kejriwal. kejriwal, who trashes every other leader, never criticises sonia. touching.
Generating Realistic Natural Language Counterfactuals
: sonia kejriwal, who .other leader criticises sonia. . PWWS+ rt @user: sonia ever criticises kejriwal ., who trashes every other leader, never criticises touching. Masked-LM rt @user: load defend extensionss ke auditionsri ore .ld charged willy rod, honest trashes prevented liu TextFooler rt @subscriptions: sonia never criticises kejriwal. kejriwal, who trashes each other executives, never criticises sonia. touching. Ours (top-1) rt @user: sonia never criticises kejriwal‚îÄ kejriwal, who trashes every other leader, never criticises sonia. touching. Ours (top-3) rt @user: sonia criticises kej wall. kejriwal, who trashes every other leader, never criticises sonia. touching b*tch. Ours (top-5) rt @user: sonia criticises kejriwal. kejriwal, who trashes every b*tch leader, never criticises sonia. b*tch touching b*tch.  (black-box BE, from positive to negative)
Generating Realistic Natural Language Counterfactuals
Original the movie has lots of dancing and fabulous music SEDC movie lots and music PWWS+ the movie lack lots of fabulous music Masked-LM the movie has 295 of dancing andial music TextFooler the photo possesses parcel of cheer and amazing symphonic Ours (top-1) the movie has loads of dancing and fabulous music Ours (top-3) the movie has loads of dancing and terrible music Ours (top-5) the movie has a loss of music and terrible music  (black-box WB, from positive to negative)
Generating Realistic Natural Language Counterfactuals
Original the problem with concept films is that if the concept is a poor one, there's no saving the movie SEDC the with concept films is that if concept is a, there no saving the PWWS+ the problem with concept films is that if the concept is a rich one, there saving the movie Masked-LM the problem with concept films is that if the concept is a ≥ one, there's no saving the movie TextFooler the matters with concepts movie is that if the concepts is a poorer one, there's no save the film Ours (top-1) the problem with concept films is that if the concept is a good one, there's no Ours (top-3) the problem with films is that if the concept is a good one, there's no saving Ours (top-5) the thing with concept films is that if the concept is a good one, there's saving SNLI (black-box WB, from neutral to contradicting where edits are only applied to the premise)
Generating Realistic Natural Language Counterfactuals
Premise: a man is posing on a ski board with snow in the background. Hypothesis: a naked man is posing on a ski board with snow in the background.
Generating Realistic Natural Language Counterfactuals
SEDC a man is on ski board snow in. PWWS+ a man is a ski board snow in the play up. Masked-LM a man is posing on a ski board with snow in the background. TextFooler a friend is parading on a slalom juries with blizzards in the wellspring. Ours (top-1) a man is posing on a ski board with snow in the background. Ours (top-3) a girl is sitting on a ski board with snow in the background. Ours (top-5) a girl is posing on a ski with snow in the background. 