What are the different methods which models intermediate explanations as latent variables when solving NLP tasks?
Explanations as latent variables. Modeling intermediate explanations as latent variables is a common approach, although training and inference details differ. Here we consider representative works in NLP. Zhou et al. (2020) apply a latent variable model to language understanding and train the model with variational expectation maximization. Their method can generate free-form explanations but requires a small set of labeled examples for supervision. Zhou et al. (2021b) apply such a model to probe dialogue generation in a zero-shot manner. Vig et al. (2020) apply a latent variable model to analyze gender bias in large pretrained language models by viewing the behaviors of neurons as unobserved explanations. Lei et al. (2016); Vafa et al. (2021) apply such a model to identify rationales
for sequence classification/generation, where rationales are a minimal subset of inputs or previous words that can lead to the same predictions. 
Probing Commonsense Explanation in Dialogue Response Generation
Humans use commonsense reasoning (CSR) implicitly to produce natural and coherent responses in conversations. Aiming to close the gap between current response generation (RG) models and human communication abilities, we want to understand why RG models respond as they do by probing RG model's understanding of commonsense reasoning that elicits proper responses. We formalize the problem by framing commonsense as a latent variable in the RG task and using explanations for responses as textual form of commonsense. We collect 6k annotated explanations justifying responses from four dialogue datasets and ask humans to verify them and propose two probing settings to evaluate RG models' CSR capabilities. Probing results show that models fail to capture the logical relations between commonsense explanations and responses and fine-tuning on in-domain data and increasing model sizes do not lead to understanding of CSR for RG. We hope our study motivates more research in making RG models emulate the human reasoning process in pursuit of smooth human-AI communication.
[{"first":"Pei","last":"Zhou","middle":[]},{"first":"Pegah","last":"Jandaghi","middle":[]},{"first":"Bill","last":"Lin","middle":["Yuchen"]},{"first":"Justin","last":"Cho","middle":[]},{"first":"Jay","last":"Pujara","middle":[]},{"first":"Xiang","last":"Ren","middle":[]}]
Rationales for Sequential Predictions
Sequence models are a critical component of modern NLP systems, but their predictions are difficult to explain. We consider model explanations though rationales, subsets of context that can explain individual model predictions. We find sequential rationales by solving a combinatorial optimization: the best rationale is the smallest subset of input tokens that would predict the same output as the full sequence. Enumerating all subsets is intractable, so we propose an efficient greedy algorithm to approximate this objective. The algorithm, which is called greedy rationalization, applies to any model. For this approach to be effective, the model should form compatible conditional distributions when making predictions on incomplete subsets of the context. This condition can be enforced with a short fine-tuning step. We study greedy rationalization on language modeling and machine translation. Compared to existing baselines, greedy rationalization is best at optimizing the sequential objective and provides the most faithful rationales. On a new dataset of annotated sequential rationales, greedy rationales are most similar to human rationales.
[{"first":"Keyon","last":"Vafa","middle":[]},{"first":"Yuntian","last":"Deng","middle":[]},{"first":"David","last":"Blei","middle":[]},{"first":"Alexander","last":"Rush","middle":[]}]
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Many interpretation methods for neural models in natural language processing investigate how information is encoded inside hidden representations. However, these methods can only measure whether the information exists, not whether it is actually used by the model. We propose a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior. The approach enables us to analyze the mechanisms that facilitate the ﬂow of information from input to output through various model components, known as mediators. As a case study, we apply this methodology to analyzing gender bias in pre-trained Transformer language models. We study the role of individual neurons and attention heads in mediating gender bias across three datasets designed to gauge a model’s sensitivity to gender bias. Our mediation analysis reveals that gender bias effects are concentrated in speciﬁc components of the model that may exhibit highly specialized behavior.
[{"first":"Jesse","last":"Vig","middle":[]},{"first":"Sebastian","last":"Gehrmann","middle":[]},{"first":"Yonatan","last":"Belinkov","middle":[]},{"first":"Sharon","last":"Qian","middle":[]},{"first":"Daniel","last":"Nevo","middle":[]},{"first":"Yaron","last":"Singer","middle":[]},{"first":"Stuart","last":"Shieber","middle":["M."]}]
Rationalizing Neural Predictions
Prediction without justification has limited applicability. As a remedy, we learn to extract pieces of input text as justifications -- rationales -- that are tailored to be short and coherent, yet sufficient for making the same prediction. Our approach combines two modular components, generator and encoder, which are trained to operate well together. The generator specifies a distribution over text fragments as candidate rationales and these are passed through the encoder for prediction. Rationales are never given during training. Instead, the model is regularized by desiderata for rationales. We evaluate the approach on multi-aspect sentiment analysis against manually annotated test cases. Our approach outperforms attention-based baseline by a significant margin. We also successfully illustrate the method on the question retrieval task.
[{"first":"Tao","last":"Lei","middle":[]},{"first":"Regina","last":"Barzilay","middle":[]},{"first":"Tommi","last":"Jaakkola","middle":[]}]
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Recently generating natural language explanations has shown very promising results in not only offering interpretable explanations but also providing additional information and supervision for prediction. However, existing approaches usually require a large set of human annotated explanations for training while collecting a large set of explanations is not only time consuming but also expensive. In this paper, we develop a general framework for interpretable natural language understanding that requires only a small set of human annotated explanations for training. Our framework treats natural language explanations as latent variables that model the underlying reasoning process of a neural model. We develop a variational EM framework for optimization where an explanation generation module and an explanation-augmented prediction module are alternatively optimized and mutually enhance each other. Moreover, we further propose an explanation-based self-training method under this framework for semi-supervised learning. It alternates between assigning pseudo-labels to unlabeled data and generating new explanations to iteratively improve each other. Experiments on two natural language understanding tasks demonstrate that our framework can not only make effective predictions in both supervised and semi-supervised settings, but also generate good natural language explanation.
[{"first":"Wangchunshu","last":"Zhou","middle":[]},{"first":"Jinyi","last":"Hu","middle":[]},{"first":"Hanlin","last":"Zhang","middle":[]},{"first":"Xiaodan","last":"Liang","middle":[]},{"first":"Maosong","last":"Sun","middle":[]},{"first":"Chenyan","last":"Xiong","middle":[]},{"first":"Jian","last":"Tang","middle":[]}]
Probing Commonsense Explanation in Dialogue Response Generation
Response generation (RG) systems, which have the basic goal of mimicking human conversation, have as of yet an unmeasured ability to understand communicative intents. In general, standard neural language models build correlative models of linguistic stimuli rather than deep understanding of humanlevel meaning (Bender and Koller, 2020). As such, there is reason to suspect that, while RG systems today have impressive performance on common metrics (Zhang et al., 2020b;Roller et al., 2021), they achieve this performance without truly understanding human communication. Commonsense reasoning (CSR), defined as "the basic level of practical knowledge and reasoning concerning everyday Figure 1: A motivating example for our study. We want to know whether RG models understand the implicit common sense that justifies dialogue responses.
Probing Commonsense Explanation in Dialogue Response Generation
situations and events that are commonly shared among most people" (Sap et al., 2020), is critical in human communication. Specifically, CSR helps establish a common ground consisting of "mutual knowledge" between participants, which is key to smooth communication (Clark and Schaefer, 1989;Clark and Brennan, 1991).
Probing Commonsense Explanation in Dialogue Response Generation
For example, consider a conversation between two friends shown in Figure 1. The reason the person on the right (responder) is happy is not indicated explicitly, but it is common sense that finding a buyer for the house (that the responder is likely aiming to sell) makes one happy, which explains the response "I'm so happy". Motivated by how humans communicate, we ask a main research question: do RG models understand the implicit CSR that explains why a response makes sense? This will help us analyze whether the RG models that seem to produce human-like responses really understand the reasoning process that justifies the response, which is important to build a reliable and robust dialogue system. Furthermore, understanding implicit common sense behind RG can also help make models generate more natural and coherent responses.To answer this important research question, we present our initial findings from annotating commonsense explanations in dialogues and evaluating RG models for commonsense reasoning capabilities.
Probing Commonsense Explanation in Dialogue Response Generation
We first present a probing setup for evaluating common sense in RG, called CEDAR: Common sEnse in DiAlogue Response generation. We start with formalizing CSR in RG by considering common sense as a latent variable that helps explain the observed variable "response" in the RG process -similar to how humans use common sense in communication (Hilton, 1990). To instantiate implicit common sense for probing, we use textual explanations of the response as the common sense embedded in the dialogue context. To understand whether RG models can comprehend implicit common sense, we corrupt explanations to break the logical coherence and compare model behaviors between a valid explanation and a corrupted one.
Probing Commonsense Explanation in Dialogue Response Generation
To operationalize the probing, we collect the first annotations on commonsense explanations that justify dialogue responses. Each annotation is a dialogue-specific explanation that explicitly describes what might cause the response in one of the five dimensions: event, emotion, location, possession, and attribute, inspired by human cognitive psychology (Kintsch and Van Dijk, 1978). We find through pilot studies that directly asking people to annotate result in explanations with high variation and subjectivity, to account for this, we first generate candidate explanations by adopting a large textto-text language model trained on a story explanation dataset, namely GLUCOSE (Mostafazadeh et al., 2020), under the dialogue setting. Next, we conduct a carefully designed two-stage human verification process with a qualification test and the main annotation task. We present our findings from verifying 6k generated explanations on 1,200 dialogues sampled from four public dialogue datasets.
Probing Commonsense Explanation in Dialogue Response Generation
Using the annotated explanations, we probe state-of-the-art (SOTA) RG models for two CSRrelated abilities: (i) the ability to understand whether the commonsense explanation can justify a response, and (ii) the ability to attribute logically-coherent explanations for dialogue responses. These are inspired by what showcases human understanding of common sense in conver-sational communication. Our probing setup contrasts valid explanations with corrupted version. Corruptions are generated via two methods: logical corruptions that disrupt logical coherence, and complete corruption where we disrupt the grammatical naturalness of the sentence.
Probing Commonsense Explanation in Dialogue Response Generation
We find that the models fail to understand common sense that elicits proper responses according to performance on our probing settings and some models even do not distinguish gibberish sentences. Fine-tuning on in-domain dialogues and verified explanations do not help with understanding. We also find interesting cases that show potential statistical biases in RG models. We hope our annotated explanations and probing findings encourage more studies on making RG models communicates with deep understanding of human reasoning process.
Probing Commonsense Explanation in Dialogue Response Generation
This section first introduces how we incorporate common sense as a latent variable in the RG setting. Then we specify two challenges that arise in order to examine whether RG models can comprehend common sense to arrive at responses similarly as humans do. Lastly, we present our solutions to the challenges by instantiating common sense as textual explanation and proposing two probing settings to evaluate if models reason about common sense when generating responses.
Probing Commonsense Explanation in Dialogue Response Generation
Preliminaries We consider the classic dialogue response generation (RG) setup (Weizenbaum, 1966;Ritter et al., 2011;Sordoni et al., 2015): given a dialogue history H, generate an appropriate response R. Most state-of-the-art (SOTA) neural RG models generate a response given a dialogue history as a conditional language modeling problem. Specifically, given a history (H) consisting of a sequence of dialogue turns from the dialogue history x 1 , x 2 , ..., x n (each containing a sequence of tokens) and a response (R) sentence y comprised of a sequence of tokens y 1 , y 2 , ..., y m , RG models aim to learn the conditional probability distribution by training on human dialogues:
Probing Commonsense Explanation in Dialogue Response Generation
(1)
Probing Commonsense Explanation in Dialogue Response Generation
Common Sense as a Latent Variable As illustrated in Figures 1 and 2, when humans respond in a conversation, we use common sense implicitly to Figure 2: Probing setting illustrations. We draw inspirations from human reasoning process during communication and probe RG models' understanding of implicit common sense in RG in two ways (red and blue dotted lines).
Probing Commonsense Explanation in Dialogue Response Generation
establish common ground (Grice, 1975;Clark and Brennan, 1991), reach mutual understanding, and help produce natural responses for smooth communication. We consider common sense to be latent because it is infrequently stated due to the cooperative principle that states that participants should "not make your contribution more informative than is required" (Grice, 1975). However, the reasoning it enables is an integral part establishing common ground and critical for communication. To formalize this process, we consider common sense (CS) as an important latent variable in the modeling of a dialogue response when given the historyi.e, P (R|H, CS). Other latent factors such as the environment in which the conversation happens and background information of the participants can also influence the dialogue, but here we focus on common sense.
Probing Commonsense Explanation in Dialogue Response Generation
Current RG models generate responses in an end-toend manner with only input from dialogue history (i.e., H), making it non-trivial to examine if they understand the implicit common sense behind RG process (also see Figure 2 for an illustration). We instantiate implicit common sense in dialogues and then design probes to evaluate models' grasp of common sense. This leads to two key challenges: 1) how to instantiate abstract and implicit common sense CS in dialogues? and 2) how to probe RG models' understanding of common sense in dialogue response generation?
Probing Commonsense Explanation in Dialogue Response Generation
We use natural language explanations justifying why a response makes common sense as a proxy to instantiate common sense in RG. Traditional studies have tied common sense and the ability to provide explanations for events and actions closely (Hansen, 1980;Hilton and Slugoski, 1986), which also holds true in a conversational setting (Hilton, 1990). Specifically, as shown in Figure 1, "I want to make sure my house is sold" is a potential explanation about what leads to "hiring a real estate agent" in the response and this explanation requires understanding the commonsense relation that a real estate agent helps sell a house and the desire to sell a house motivates a person to hire an agent. Formally, we concretize the abstract latent variable common sense CS in textual form as an explanation E explaining what might cause the response R given the history H. We introduce our process of collecting such explanations for RG in Section 3.
Probing Commonsense Explanation in Dialogue Response Generation
We then draw inspiration from human reasoning process behind dialogue response generation to design two probing tasks. First, humans use common sense implicitly to produce natural and coherent responses in conversations (Clark and Schaefer, 1989). Common sense helps humans determine what responses make sense in certain context. We want to see if providing common sense in the form of explanation also helps RG models arrive at coherent and natural responses more easily. Second, humans can perform causal attribution on an event or an action by finding reasons that might cause it (Hilton, 1990). If the person producing the response is asked about why they are feeling happy, they can easily respond with reasons about their reasoning process. We are interested in examining can RG models also generate responses to justify a previous response when asked. We probe RG models in a contrastive manner, by comparing model behaviors with a valid explanation E to the response and a corrupted E that breaks logical coherence. We introduce the two settings in more detail as follows.
Probing Commonsense Explanation in Dialogue Response Generation
Inference Probing Here we directly measure if P (R|E, H) > P (R|E , H) for RG models, i.e., can models assign a higher probability to the response when provided with valid common sense in the form of explanations compared to logicallyincoherent explanations? Since existing RG models are not trained to take explanations as additional input, the probing results may be confounded by the model's unfamiliarity with the probing set-ting. To account for this issue, we 1) probe on a knowledge-grounded RG model that is used to taking in additional knowledge sentences as input and 2) fine-tune RG models on a proportion of our collected explanation and compare the effects. We discuss results and issues about probing models fine-tuned on explanations in Section 5.3.
Probing Commonsense Explanation in Dialogue Response Generation
If the model assigns a similar or lower probability to the response given a valid explanation compared to a logically-incoherent explanation, it indicates that the reason why this response makes sense is not clear for models.
Probing Commonsense Explanation in Dialogue Response Generation
Attribution Probing Here we examine if P (E|H, R) > P (E |H, R), i.e., can RG models perform causal attribution as humans by assigning a higher probability to a valid explanation of the response (that makes sense) compared to a corrupted explanation, given the dialogue history and the response? To address the unfamiliarity of models, we make the probing setting close to real dialogues by continuing the conversation (consisting of H and R) with "why" to prompt the models to generate an explanation. We also conduct fine-tuning on a proportion of our collected explanations similarly to the first setting discussed in Section 5.3.
Probing Commonsense Explanation in Dialogue Response Generation
If the model prefers the attribution of the response that is incoherent with the response by giving it a higher probability, it indicates that the model fails to generate valid reasons for responses, which requires understanding the implicit common sense behind dialogues.
Probing Commonsense Explanation in Dialogue Response Generation
To get explanation annotations for dialogue responses, we first automatically generate commonsense causal explanations and then manually verify via crowdsourcing. We use a text-to-text model trained on commonsense story explanation dataset GLUCOSE (Mostafazadeh et al., 2020) as the generator and conduct 2-stage human verification on generated explanations. We first introduce the model we use, the adaptation of the model on dialogue data, and our verification process to ensure the quality of generated explanations.
Probing Commonsense Explanation in Dialogue Response Generation
GLUCOSE is a large-scale dataset of implicit commonsense causal explanations grounded in a story context (Mostafazadeh et al., 2020). Given a Figure 3: Examples of human-verified commonsense explanations for the dialogue shown in Figure 1 short story and a sentence X in the story, GLU-COSE contains human annotations of five dimensions of causal explanation related to X (an event/emotion/location/possession/attribute leads to X), each in a semi-structured form "antecedent connective consequent." Using the collected explanations, the authors train state-of-the-art neural models and find that the trained models are able to produce commonsense inferences on unseen stories. More details about how models are trained on GLUCOSE are included in Appendix A.
Probing Commonsense Explanation in Dialogue Response Generation
We consider using a model trained on GLU-COSE to automatically generate commonsense explanations in dialogues for several reasons. First, it generates contextual commonsense explanations that provides causal knowledge about what justifies a sentence. Second, it provides fine-grained causal explanations along different dimensions. Last but not least, we have conducted multiple rounds of pilot studies to directly ask workers to write out commonsense explanations for a response, but the subjectivity of this open-ended task led to large variations in quality. Instead we ask workers to verify explanations generated from a model.
Probing Commonsense Explanation in Dialogue Response Generation
We sample 1,200 dialogues from 4 dialogue datasets (300 from each): DailyDialog (Li et al., 2017), EmpatheticDialogues (Rashkin et al., 2019), MuTual (Cui et al., 2020), and SocialIQAprompted dialogues (Zhou et al., 2021). We generate 6k commonsense causal explanations (5 dimensions for each dialogue), using the last turn as the response and the previous turns as dialogue history (after filtering short turns). We follow Zhou et al. (2021)'s approach to select dialogues that contain at least a one-hop triple from ConceptNet (Liu and Singh, 2004). We use the same hyperparameters and weights from the best-performing 770M T5 model from Mostafazadeh et al. (2020).
Probing Commonsense Explanation in Dialogue Response Generation
To ensure the quality of generated explanations, we carefully design a two-stage human verification process with a qualification test and the main task.
Probing Commonsense Explanation in Dialogue Response Generation
Workers must first pass a qualification test (QT) that tests their understanding of the CS criteria necessary for our main annotation tasks (more details in Appendix B). We consider three criteria, requiring generated explanations to pass all three to be considered a valid commonsense explanation for a response. We ask three workers on Amazon Mechanical Turk (MTurk) to annotate the three criteria for each explanation.
Probing Commonsense Explanation in Dialogue Response Generation
Criteria 1). Relevant. A good causal explanation has to focus on explaining what could cause the response in the dialogue context (Hilton, 1990). An example of an irrelevant explanation for the example shown in Figure 1 is "I possess a house enables I live in a house" since "living in a house" is not what the response is about, so it doesn't help explain the response. 2). Non-trivial. We observe that sometimes the model simply duplicates a previous dialogue turn as the cause, which trivially associates history and response. We are interested in implicit and specific commonsense so we filter out explanations that parrot a previous turn. For example, "I found a buyer for the house motivates Oh Boy! I'm so happy. I knew hiring a real estate agent was a good idea." 3). Plausible. We ask humans to verify if the generated explanation plausibly identifies a likely cause for the response. An example of an implausible explanation is "I am in a house enables I am so happy" since "I am in a house" is not the direct cause why the person producing the response is feeling so happy, "found a buyer for the house" is. This is the hardest criterion for humans to decide due to its subjectivity nature.
Probing Commonsense Explanation in Dialogue Response Generation
We present results of our verification of 6k explanations from three in-house annotators. To filter ambiguous explanations and be strict about the quality of verified explanations, we only consider explanations valid if all three annotators have agreed that they satisfy all three criteria, i.e., 100% agreement for all verified explanations. For the annotated explanation, passing rates (agreed by 3 workers) for criterion (relevant, non-trivial, plausible) are (55%, 73%, 37%) -yielding an overall passing rate of 26% (1,560 explanations). Passing rates for the five dimensions (event, emotion, location, possession, attribute) are (31%, 33%, 13%, 24%, and 29%), with more details in Appendix B. Figure 3 presents examples for different dimension, full data is included in the supplementary material. 
Probing Commonsense Explanation in Dialogue Response Generation
We probe RG models' capability of understanding and using the explanation E in a contrastive manner (Sec. 2.2). This section first introduces the corruption types under two categories, then we introduce evaluation metrics, and finally we discuss several SOTA RG models with different neural architectures that we probe.
Probing Commonsense Explanation in Dialogue Response Generation
We use verified explanations generated from GLU-COSE T5 model as valid explanations and define two categories of corruptions to corrupt the explanation to be logically-invalid and/or grammatically unnatural. We consider three logical corruptions that invalidate the logical connection between explanation and response, as well as three complete corruptions that break both logical coherence and naturalness of the sentence. Examples covering showing corruption types of a valid explanation are shown in Figure 4, for which "I found a buyer for the house" is the antecedent, "causes" is the connective, and "I am so happy" is the consequent.
Probing Commonsense Explanation in Dialogue Response Generation
We consider three ways to invalidate the logic of the explanation: 1) Swapped that swaps the antecedent and consequent of the explanation, 2) Negation that negates the connective word of the explanation, 3) Incorrect that uses an explanation from the same dialogue historyresponse instance that is rated as incorrect (if any) during the verification.
Probing Commonsense Explanation in Dialogue Response Generation
Complete Corruptions Inspired by Sankar et al. (2019) who design perturbations to apply on dialogue history and analyze sensitivity of RG models by measuring the perplexity of the response, we consider three operations that completely break the naturalness of the explanation: 1) Shuffle that randomly shuffles the words of the explanation, 2) Dropped that drops 30% of the words uniformly, 3)   Table 1: Inference probing results for different response generation models on 4 dialogues datasets against two categories of corruptions. Accuracy is the bianry accuracy of giving a lower loss to the valid explanation than the corrupted one and ∆ NLL is the average difference of per-token NLL between the loss of a corrupted inference and a valid inference (the more positive the better).
Probing Commonsense Explanation in Dialogue Response Generation
Reversed reverses the ordering of all the words in the explanation.
Probing Commonsense Explanation in Dialogue Response Generation
We use two metrics to measure RG models' capability to distinguish valid commonsense causal explanations from invalid explanations. The standard way of modeling P θ in Equation (1) in generative models is using Maximum Likelihood Estimation (MLE) approach and minimize the conditional negative log-likelihood loss (NLL), i.e., L(P θ , R, H) = − m i=1 log P θ (y i |y <i , x 1 , ..., x n ). Since NLL is a direct measure of the probability distribution learned by the models, we use the same NLL measure for probing RG models' behavior. To measure performance of RG models, we directly compare the average per-token NLL when given a valid explanation and when given an invalid explanation to the response.
Probing Commonsense Explanation in Dialogue Response Generation
We first consider binary accuracy of giving a lower loss (higher probability) to the valid explanation than the corrupted one. A random-guessing baseline for the accuracy is 0.5. To further measure how confident the model is in determining the validity of commonsense explanations, we also compute the average difference ∆N LL by subtracting the loss of the valid inference from the invalid inference loss. The closer to zero the difference is, the less confident the model is.
Probing Commonsense Explanation in Dialogue Response Generation
We experiment with multiple models from two neural architectures: GPT-2-based (Radford et al., 2019) unidirectional transformer language model and Seq2Seq-based transformer (Vaswani et al., 2017) models. For GPT-2-based models, we use DialoGPT that is trained on 147M multi-turn conversation-like exchanges extracted from Red-dit (Zhang et al., 2020b) and GPT-2 trained on TopicalChat (Gopalakrishnan et al., 2019) as the knowledge-grounded RG model. For seq2seq models, we use BlenderBot (Roller et al., 2021) and BART (Lewis et al., 2020). More details about these RG models are included in Appendix C.
Probing Commonsense Explanation in Dialogue Response Generation
We present results and findings for our two probing settings using different dialogue RG models across four datasets for which we collected verified explanations. For each human-validated explanation, we generate a corrupted version using one of our six corruption types, and compare the NLL for the probe target according to our two settings.
Probing Commonsense Explanation in Dialogue Response Generation
In Tables 1 and 2, we show both binary accuracy and average difference in NLL for dialogues from four datasets under the two settings and aggregate the six corruption types into two categories. We also sample 5% of the dialogues for human verification under the same two probe settings.
Probing Commonsense Explanation in Dialogue Response Generation
Change Given Explanations?
Probing Commonsense Explanation in Dialogue Response Generation
All models are insensitive to the relation between explanations and responses. As shown in the left portion of Table 1, we find that when comparing a valid explanation with a logically corrupted (LC) one, all models, regardless of left-toright or seq2seq model architecture, have accuracy around 50-60%, near a random guessing baseline, with extremely small differences in NLL (some even negative). This suggests that the RG models do not understand the causal relation between the explanation and the response since they give similar probabilities to the response when conditioned on a valid explanation and on a incoherent expla-  nation, while humans can easily identify the valid explanation.
Probing Commonsense Explanation in Dialogue Response Generation
Even gibberish does not change response probability much. Surprisingly, we find that even when corrupting the explanation so completely that it becomes unnatural English, most seq2seq RG models still generate responses with a roughly equal likelihood (left2right models perform better but still lag human performance) as shown in the right portion of Table 1. Sankar et al. (2019) find that the increase in perplexity of the response is tiny when they perturb the dialogue context, but here we find that there might even not be any increase in perplexity when conditioned on gibberish compared to a valid explanation expressed in English, while humans can identify the natural explanation perfectly.
Probing Commonsense Explanation in Dialogue Response Generation
Logically incoherent attribution confuses the models. Similar to the inference probing setting, for logically corrupted one, all models have accuracy around 50-60% and tiny differences in NLL from the left part of Table 2. This indicates that the RG models cannot identify a logically-valid reason for a response from a reason that is similarly natural in terms of grammar but with totally different and invalid logical implications for the dialogue. Humans, from our sampled dialogues, again show much higher accuracy in this setting.
Probing Commonsense Explanation in Dialogue Response Generation
Models can confidently distinguish valid attribution from unnatural ones. For complete corruptions (CC), we find that except for BART, RG models perform much better in identifying a valid explanation compared to a completely corrupted one with most accuracy being close to 1 and rel-  Figure 5: Model size Effects on the two probing settings for BART aggregated across four datasets and types of corruptions. We find that except for the attribution setting against complete corruptions, increasing size does not impact much on probing performance. atively larger NLL differences. We conclude that these RG models find generating a valid explanation more natural than completely corrupted ones, which is expected since they are trained to generate natural sentences. However, combining this finding with the previous observation, we find that these RG models can discern unnatural sentences by giving a low probability, but fail to determine the logical validity of the reasons for responses, posing doubts on whether they understand CSR behind a response.
Probing Commonsense Explanation in Dialogue Response Generation
Unfamiliarity with probing format is not the bottleneck. Since these RG models are not trained directly to take additional knowledge as input to generate responses or generate explanations for responses (although explaining happens often in dialogues), these poor results may be due to the probing setup. We thus fine-tune BART-base on 50% of our verified explanations in the same format as our two settings and probe on the rest. We find even when the model is accustomed to the : Fine-tuning (on in-domain dialogues) Effects on the two probing settings for three models aggregated across four datasets and types of corruptions. We find that in general fine-tuning does not help and sometimes even hurt the probing performance.
Probing Commonsense Explanation in Dialogue Response Generation
tasks, the accuracy against logical corruptions for both settings is still around 60%. Although it is possible that with more data the performance can be improved, we also note that training with explanations also makes the model biased to prefer explanations over corrupted ones due to pattern matching. For example, no explanations contain negated connectives, which might be used to gain an advantage unrelated to understanding common sense when compared against negated corruption. To probe a model that is accustomed to the task but not exposed to explanation patterns, we consider a GPT-2-based (Radford et al., 2019) model trained on TopicalChat (Gopalakrishnan et al., 2019), a knowledge-grounded dialogue dataset. The model is trained on given input of dialogue history concatenated with a knowledge sentence that the response needs to use. We treat the commonsense explanation as the knowledge sentence as they both provide necessary information that leads to the response. We find that the model performs similarly to DialoGPT on our probing setting for logical corruptions, providing evidence that the reason why these RG models cannot identify causal relations behind dialogue responses is not because the model is not used to taking explanation as input.
Probing Commonsense Explanation in Dialogue Response Generation
Model size does not help with understanding common sense. Comparing BART-base and BART-large in Figure 5, we find that except for the attribution setting with complete corruptions, size does not change probing results (even lower accuracy against logical corruptions), indicating that the size of RG model is not the key to understand commonsense explanations for dialogue responses. Fine-tuning on in-domain dialogues sometimes have opposite effects. Since these RG models are trained on different dialogue datasets that are not necessarily in the same domain as probing dialogues, we also explore the effects of fine-tuning on in-domain dialogues, dialogues from the 4 datasets we use for probing. Three pairs of model (before and after fine-tuning) results are shown in Figure 6 and we do not find significant differences. We even find sometimes fine-tuning hurts the probing results, which might be due to models picking up statistical patterns while training on similar dialogues, relying less on "reasoning", if any.
Probing Commonsense Explanation in Dialogue Response Generation
Potential biases on certain perturbation types.
Probing Commonsense Explanation in Dialogue Response Generation
The observations above are general trends of the models performance, but we also find interesting corner cases indicating potential biases in the models when we breakdown performance for six corruption types shown in Figure 7 on the attribution probing setting. For the Negated corruption type, DialoGPT and BART have accuracy around 30%, meaning that for 70% of the time, they prefer generating explanations with negated relations in it.
Probing Commonsense Explanation in Dialogue Response Generation
We study commonsense reasoning in dialogue response generation aiming to close the gap between current RG models and human communication abilities. Specifically we formalize the problem by framing commonsense as a latent variable in the RG task and using explanations for responses as textual form of commonsense. We design an explanation collection procedure for RG and propose two probing settings to evaluate RG models' CSR capabilities. We hope our study motivates more research in making RG models emulate human reasoning process in pursuit of smooth human-AI communication.
Probing Commonsense Explanation in Dialogue Response Generation
Our work aims to examine RG model's ability to understand common sense for dialogue responses. Sheng et al. (2021) have found biases in DialoGPT responses and Mehrabi et al. (2021) have found representational harms in common sense resources.
Probing Commonsense Explanation in Dialogue Response Generation
We acknowledge that the generated responses from models we use in probing experiments might contain biases. All of the dialogue datasets and models are in English, which benefits English speakers more. We have conducted human verification using Amazon Mechanical Turks. We pay turkers around $14 per hour, well above the highest state minimum wage and engage in constructive discussions if they have concerns about the process. We also give each annotation instance enough time so that we do not pressure annotators.
Probing Commonsense Explanation in Dialogue Response Generation
GLUCOSE contains human annotations of ten dimensions of causal explanation related to X. Five of the dimensions are about events and states happening before X and five are about those happening after X. Specifically, inspired by cognitive psychology, the authors of GLUCOSE consider events, emotions, location states, possession states, and other attributes as the five dimensions of causal inferences. According to their evaluation, the bestperforming model is T5 (Raffel et al., 2020) (with 770M parameters) with the input formulated as #d : S * [X], where d is the dimension and S * [X] is the story S with sentence X surrounded by asterisks". An illustrated example of inputs of outputs of the T5 model trained on GLUCOSE is shown in Figure 8.
Probing Commonsense Explanation in Dialogue Response Generation
To adopt the T5 model trained on GLUCOSE to our task: generating explanations about what might cause producing a response given a dialogue history, we append the dialogue history turns together, enclose the response we are interested in explaining with asterisks, and fill in dimension number 1 to 5 to ask for what event, emotion, location, possession, and attribute could cause, motivate, or enable the response. In other words, we formulate our queries as #d : H * [R], where d is the dimension 1 to 5 and H * [R] is the dialogue history H appended with the response R surrounded with asterisks. Table 3 shows the general pass rate for each criterion and the overall pass rate (need to pass all three criteria). Figure 9 shows distribution of valid and invalid explanations separated by the five causal dimensions. We find the explanations about a location state that causes the response have a lower valid rate (13%) than others. This might be due to that in some dialogues the location information is not important in explaining the response and thus it is difficult to come up with a plausible reason about a location that leads to the response. All other dimensions have a similar rate of 25-30%.
Probing Commonsense Explanation in Dialogue Response Generation
To ensure annotation quality, the workers first need to pass a qualification test (QT) that tests their understandings of the criteria to be able to do our main annotation tasks. Our QT contains eight questions, each contains a dialogue history, a response, and an explanation and we ask them to choose whether this explanation satisfies a specific criterion from the three above. The eight questions are formed into 4 pairs each consisting of a training question and a testing question and each pair focuses on the same criterion. For the relevance and non-trivial criteria, we have one pair for each and for the plausible criterion, we have two pairs since it is trickier to determine than the other two. We provide the right answer with explanation for the training question whether they answer it correctly or not and use the testing questions for assessment of their understanding.
Probing Commonsense Explanation in Dialogue Response Generation
DialoGPT extends the GPT-2 architecture that adopts the generic transformer language model (Vaswani et al., 2017)     model 2 (Zhang et al., 2020b).
Probing Commonsense Explanation in Dialogue Response Generation
BlenderBot is proposed by Roller et al. (2021) using a standard seq2seq transformer architecture (Vaswani et al., 2017). The model aims to blend multiple conversational skills. Human evaluations show their best models beat existing approaches in multi-turn dialogue in terms of engagingness and humanness. We use the 400M Blender-Bot model distilled from 2.7B parameter model 3 .
Probing Commonsense Explanation in Dialogue Response Generation
BART is proposed by Lewis et al. (2020) using a standard seq2seq architecture with a bidirectional BERT encoder and a left-to-right GPT decoder. It uses denoising pre-training objectives and has shown to outperform previous models in multiple  Figure 12: Dataset differences on the causality probing setting for DialoGPT model, we find that there is no drastic differences in probing performances across four datasets for logical corruptions, i.e., the conclusion that RG model fails to understand causality holds true for all datasets. We see difference in accuracy ranging from 70% to 90% for complete corruptions.
Rationales for Sequential Predictions
Sequence models are a critical component of generation tasks ranging from language modeling to machine translation to summarization. These tasks are dominated by complex neural networks. While these models produce accurate predictions, their decision making processes are hard to explain. Interpreting a model's prediction is important in a variety of settings: a researcher needs to understand a model to debug it; a doctor using a diagnostic model requires justifications to validate a decision; a company deploying a language model relies on model explanations to detect biases appropriated from training data.
Rationales for Sequential Predictions
Interpretation takes many flavors (Lipton, 2018). We focus on rationales, i.e. identifying the most  . Rationales for sequential prediction on GPT-2. Each row is a predicted word. The dark cells correspond to the context words found by greedy rationalization. To predict "constitutionality", the model only needs "Supreme", "Court", "challenge", and "the". important subset of input tokens that leads to the model's prediction. For example, consider the sentence: "The Supreme Court on Tuesday rejected a challenge to the constitutionality of the death penalty." Suppose we would like to explain the decision of the model to generate "constitutionality". While the model mathematically conditions on all the previous words, only some are necessary for its predictions. In this case, the rationale produced by our algorithm includes "the", "challenge", and notably "Supreme Court", but not phrases that add no information like "on Tuesday" (Figure 1).
Rationales for Sequential Predictions
Various rationale methods have been proposed for sequence classification, where each sequence has a single rationale (Lei et al., 2016;Chen et al., 2018;Jain et al., 2020). However, these methods cannot scale to sequence models, where each token in a sequence requires a different rationale.
Rationales for Sequential Predictions
This work frames the problem of finding se-quence rationales as a combinatorial optimization: given a model, the best rationale is the smallest subset of input tokens that would predict the same token as the full sequence. Finding the global optimum in this setting is intractable, so we propose greedy rationalization, a greedy algorithm that iteratively builds longer rationales. This approach is efficient for many NLP models such as transformers (Vaswani et al., 2017). Moreover, it does not require access to the inner workings of a model, such as gradients.
Rationales for Sequential Predictions
Underlying this approach is an assumption that the model forms sensible predictions for incomplete subsets of the input. Although we can pass in incomplete subsets to neural models, there is no guarantee that their predictions on these subsets will be compatible with their predictions on full contexts (Arnold and Press, 1989). We show that compatibility can be learned by conditioning on randomly sampled context subsets while training a model. For large pretrained models like GPT-2 (Radford et al., 2019), fine-tuning is sufficient.
Rationales for Sequential Predictions
In an empirical study, we compare greedy rationalization to various gradient-and attention-based explanation methods on language modeling and machine translation. Greedy rationalization best optimizes the objective, and its rationales are most faithful to the inner workings of the model. We additionally create a new dataset of annotated rationales based on the Lambada corpus (Paperno et al., 2016). We find that greedy rationales are most similar to human annotations, both on our dataset and on a labeled dataset of translation alignments.
Rationales for Sequential Predictions
Our code and annotated dataset are available. 1
Rationales for Sequential Predictions
Consider a sequence of tokens, y 1:T , generated by some unknown process y 1:T ∼ F . The goal of sequence modeling is to learn a probabilistic model p θ that approximates F from samples. Maximumlikelihood estimation is an effective way to train these models, where θ is fit according to arg max θ E y 1:T ∼F [log p θ (y 1:T )].
Rationales for Sequential Predictions
(1)
Rationales for Sequential Predictions
Sequence models are typically factored into conditional distributions:
Rationales for Sequential Predictions
p θ (y 1:T ) = f θ (y 1 ) T t=2 f θ (y t |y <t ).
Rationales for Sequential Predictions
Here, f θ is the specific model parameterizing p θ , such as a transformer (Vaswani et al., 2017), and is trained to take inputs y <t . Going forward, we drop the dependence on θ in the notation. Word-level explanations are a natural way to interpret a sequence model: which words were instrumental for predicting a particular word? Would the same word have been predicted if some of the words had been missing?
Rationales for Sequential Predictions
Explanations may be straightforward for simpler models; for example, a bigram Markov model uses only the previously generated word to form predictions. However, the most effective sequence models have been based on neural networks, whose predictions are challenging to interpret (Lipton, 2018).
Rationales for Sequential Predictions
Motivated by this goal, we consider a sequence y 1:T generated by a sequence model p. At each position t, the model takes the inputs in the context y <t and uses them to predict y t . We are interested in forming rationales: subsets of the contexts that can explain the model's prediction of y t . 2 What are the properties of a good rationale? Any of the contextual words y <t can contribute to y t . However, if a model makes the same prediction with only a subset of the context, that subset contains explanatory power on its own. A rationale is sufficient if the model would produce the same y t having seen only the rationale (DeYoung et al., 2020). While rationales consisting of the full context would always be sufficient, they would be ineffective for explaining longer sequences. Intuitively, the smaller the rationale, the easier it is to interpret, so we also prioritize brevity.
Rationales for Sequential Predictions
We combine these desiderata and frame finding rationales as a combinatorial optimization: the best rationale of a word y t is the smallest subset of inputs that would lead to the same prediction. Each candidate rationale S is an index set, and y S denotes the subset of tokens indexed by S. Denote by S = 2 [t−1] the set of all possible context subsets. An optimal rationale is given by
Rationales for Sequential Predictions
The constraint guarantees sufficiency, and the objective targets brevity. Although the objective may have multiple solutions, we only require one.
Rationales for Sequential Predictions
Optimizing Equation 3 is hindered by a pair of computational challenges. The first challenge is that solving this combinatorial objective is intractable; framed as a decision problem, it is NPhard. We discuss this challenge in Section 3. The second challenge is that evaluating distributions conditioned on incomplete context subsets p(y t |y S ) involves an intractable marginalization over missing tokens. For now we assume that f (y t |y S ) ≈ p(y t |y S ); we discuss how to enforce this condition in Section 4.
Rationales for Sequential Predictions
We propose a simple greedy algorithm, greedy rationalization, to approximate the solution to Equation 3. The algorithm starts with an empty rationale. At each step, it considers adding each possible token, and it selects the one that most increases the probability of y t . This process is repeated until the rationale is sufficient for predicting y t . Figure 2 provides an overview.
Rationales for Sequential Predictions
Here is the algorithm. Begin with a rationale
Rationales for Sequential Predictions
At each step, we iteratively add a single word to the rationale, choosing the one that maximizes the probability of the word y t :
Rationales for Sequential Predictions
p(y t |y S (n) ∪k ). (5) We continue iterating Equation 5 until arg max y t p(y t |y S (n) ) = y t . The procedure will always converge, since in the worst case, S (t−1) contains the full context. The greedy approach is motivated by approximations to the set cover problem (Chvatal, 1979). In our setting, each set is a single context token, and a rationale "covers" a sequence if it results in predicting the same token.
Rationales for Sequential Predictions
This procedure is simple to implement, and it is black-box: it does not require access to the inner workings of a model, like gradients or attention.
Rationales for Sequential Predictions
While greedy rationalization can be applied to any model, greedy rationalization is particularly effective for set-based models such as transformers. If we assume the rationale size m = |S| is significantly shorter than the size of the context t, greedy rationalization requires no extra asymptotic complexity beyond the cost of a single evaluation.
Rationales for Sequential Predictions
For transformers, the complexity of each evaluation f (y t |y <t ) is quadratic in the input set O(t 2 ).
Rationales for Sequential Predictions
Each step of greedy rationalization requires evaluating f (y t |y S ), but y S can be significantly smaller than y <t . A rationale of size m will require m steps of O(t) evaluations to terminate, resulting in a total complexity of O(m 3 t). As long as m = O(t 1/3 ), greedy rationalization can be performed with the same asymptotic complexity as evaluating a transformer on the full input, O(t 2 ). In Appendix C, we verify the efficiency of greedy rationalization.
Rationales for Sequential Predictions
Greedy rationalization requires computing conditional distributions p(y t |y S ) for arbitrary subsets S. Using an autoregressive model, this calculation requires marginalizing over unseen positions. For example, rationalizing a sequence y 1:3 requires evaluating the candidate rationale p(y 3 |y 1 ), which marginalizes over the model's predictions:
Rationales for Sequential Predictions
Given the capacity of modern neural networks, it is tempting to pass in incomplete subsets y S to f and evaluate this instead as f (y t |y S ) ≈ p(y t |y S ).
Rationales for Sequential Predictions
However, since f is trained only on complete feature subsets y <t , incomplete feature subsets y S are out-of-distribution (Hooker et al., 2019). Evaluating f (y 3 |y 1 ) may be far from the true conditional p(y 3 |y 1 ). In Figure 4, we show that indeed language models like GPT-2 produce poor predictions on incomplete subsets.
Rationales for Sequential Predictions
Ideally f (y t |y S ) approximates p(y t |y S ), a property known as compatibility (Arnold and Press, 1989). Since training with Equation 1 only evaluates f on complete contexts y <t , its behavior on incomplete contexts y S is unspecified. Instead, compatibility can be obtained by training to maximize
Rationales for Sequential Predictions
where S ∼ Unif(S) indicates sampling word subsets uniformly at random from the power set of all possible word subsets, and S <t denotes the indices in S that are less than t. Jethani et al. (2021) show that the optimum of Equation 6 is the distribution whose conditional distributions are all equal to the ground-truth conditionals.
Rationales for Sequential Predictions
We approximate Equation 6 with word dropout. In practice, we combine this objective with standard MLE training to learn compatible distributions  Figure 2. One step of greedy rationalization. In (a), the rationale so far is a single word, "dogs". In (b), each candidate token is considered and "loud" results in the best probability for "bark". In (c), the token "loud" is added to the rationale. This process repeats until the most likely word is the model prediction. is heavily skewed towards contexts containing half the words in the sequence. To alleviate this problem, we modify the word dropout distribution to sample subsets of varying lengths; see Appendix D.
Rationales for Sequential Predictions
The intuition for Equation 6 is straightforward: if the model sees incomplete contexts while training, it can approximate arbitrary incomplete distributions. Since f (y t |y S ) approximates F (y t |y S ) and f (y t |y <t ) approximates F (y t |y <t ), all the conditional distributions are compatible.
Rationales for Sequential Predictions
To demonstrate the impact of training with the compatibility objective in Equation 6, we consider a synthetic majority-class language over binary strings of 19 tokens. The first 17 are sampled uniformly from {0, 1}, and the 18th token is always '='. The 19th token is 0 if there are more 0's than 1's in the first 17 tokens, and 1 otherwise.
Rationales for Sequential Predictions
We train two models: one using the standard objective in Equation 1, the other using word dropout to optimize Equation 6. Although both models have the same heldout perplexity on the full context, training with Equation 6 is required to form compatible predictions on incomplete subsets. In Figure 3, we provide both models with random sub-  . Fine-tuning GPT-2 for compatibility removes pathological repeating on incomplete contexts. For a position t, the vertical axis gives f (y t+1 = "the"|y t = "the").
Rationales for Sequential Predictions
sets S and calculate each model's probability that the last token is 1. A model that has only seen a few tokens should be less confident about the prediction of the final majority class, yet models trained without word dropout ignore this uncertainty. Models do not need to be trained from scratch with Equation 6. A model can be pre-trained with Equation 1, after which it can be fine-tuned for compatibility. As an example, when GPT-2 is not trained with word dropout, it makes insensible predictions for out-of-distribution sequences. For a sequence that contains only the token "the", GPT-2 is trained to give reasonable predictions for p(y 2 |y 1 = "the"). But when it has only seen the token "the" somewhere besides the first position of the sequence, the top prediction for the word after "the" is also "the". 3 Of course, following "the" with "the" is not grammatical. Fine-tuning for compatibility alleviates this problem ( Figure 4).
Rationales for Sequential Predictions
Finally, we find that that fine-tuning for compatibility does not hurt the heldout performance of the complete conditional distribution of each fine-tuned model (see Appendix D).
Rationales for Sequential Predictions
In this section, we go over related rationalization approaches developed for classification and discuss why they cannot scale to sequence models. We also show that the combinatorial rationale objective in Equation 3 is a global solution to a classification rationale-style objective.
Rationales for Sequential Predictions
In classification problems, a sequence x 1:T is associated with a label y. Rationale methods are commonly used in this setting (Lei et al., 2016;Chen et al., 2018;Yoon et al., 2018;Bastings et al., 2019;Jain et al., 2020;Jethani et al., 2021). The most common approach uses two models: one, a selection model q(S|x 1:T ), provides a distribution over possible rationales; the other, the predictive model p(y|x S ), makes predictions using only samples from the former model. Typically, p and q are both optimized to maximize
Rationales for Sequential Predictions
Here, F is the ground truth, unknown data distribution, and λ is a regularizing penalty that encourages smaller rationales.
Rationales for Sequential Predictions
In practice, it is infeasible to adopt this objective for sequence models. Equation 7 is centered on providing classification models with only the words in a sequence's rationale. In sequential settings, each word has a different rationale. Since sequence models make T predictions per sequence and are trained by sharing all T word representations, each token would be indirectly exposed to words in the rationales of the words it is allowed to use. A remedy would be to train sequence models without sharing representations, but this is computationally infeasible; it requires O(T 3 ) computations per sequence for transformer architectures.
Rationales for Sequential Predictions
Most classification rationale methods treat q(S|x 1:T ) as a probability distribution over all possible rationales. However, the q that maximizes Equation 7 is deterministic for any p. To see this, note that q does not appear inside the expectation in Equation 7, so it can place all its mass on a single mode. We provide a formal justification in Appendix B.
Rationales for Sequential Predictions
Since the optimal selection model q is a pointmass, the optimal rationale can be written as
Rationales for Sequential Predictions
This optimization is identical to the combinatorial optimization in Equation 3, albeit with a soft constraint on the rationale's prediction: the true label y is not required to be the maximum of p(y |x S ).
Rationales for Sequential Predictions
In practice, this soft constraint sometimes results in empty rationales (Jain et al., 2020). Since we view sufficiency as a key component of a good rationale, Equation 3 imposes a hard constraint on the rationale's prediction.
Rationales for Sequential Predictions
There are two goals in our empirical studies. The first is to compare the ability of greedy rationalization to other approaches for optimizing the combi-natorial objective in Equation 3. The second is to assess the quality of produced rationales.
Rationales for Sequential Predictions
We measure the quality of rationales using two criteria: faithfulness and plausibility. An explanation is faithful if it accurately depicts how a model makes a decision (Jacovi and Goldberg, 2020); an explanation is plausible if it can be understood and interpreted by humans (Wiegreffe and Pinter, 2019). Although sufficiency is a standard way to measure faithfulness (DeYoung et al., 2020), all the rationales that satisfy the constraint of Equation 3 are sufficient by definition. To measure plausibility, we compare rationales to human annotations. Since there do not exist language modeling datasets with human rationales, we collected annotations based on Lambada (Paperno et al., 2016). The annotated dataset is available online, along with the code used for all experiments. 4 We compare greedy rationalization to a variety of gradient-and attention-based baselines (see Section 6). To form baseline sequential rationales, we add words by the order prescribed by each approach, stopping when the model prediction is sufficient. The baselines are: l 2 gradient norms of embeddings ( To compare rationale sets produced by each method to those annotated by humans, we use the set-similarity metrics described in DeYoung et al. (2020): the intersection-over-union (IOU) of each rationale and the human rationale, along with the token-level F1, treating tokens as binary predictions (either in the human rationale or out of it).
Rationales for Sequential Predictions
We use transformer-based models for all of the experiments. We fine-tune each model for compatibility using a single GPU. That we can fine-tune GPT-2 Large (Radford et al., 2019) to learn compatible conditional distributions on a single GPU suggests that most practitioners will be able to train compatible models using a reasonable amount of computation. For model and fine-tuning details, refer to Appendix D.
Rationales for Sequential Predictions
The experiments test sequential rationales for language modeling and machine translation. Appendix E contains full details for each experiment.
Rationales for Sequential Predictions
Long-Range Agreement. The first study tests whether rationales for language models can capture long-range agreement. We create a template dataset using the analogies from Mikolov et al. (2013). This dataset includes word pairs that contain either a semantic or syntactic relationship. For each type of relationship, we use a predefined template. It prompts a language model to complete the word pair after it has seen the first word.
Rationales for Sequential Predictions
For example, one of the fifteen categories is countries and their capitals. We can prompt a language model to generate the capital by first mentioning a country and then alluding to its capital. To test long-range agreement, we also include a distractor sentence that contains no pertinent information about the word pair. For example, our template for this category is, When my flight landed in Japan, I converted my currency and slowly fell asleep. (I had a terrifying dream about my grandmother, but that's a story for another time). I was staying in the capital, _______________ Here, the parenthetical clause is a distractor sentence, since it contains no relevant information about predicting the capital of Japan. The correct capital, "Tokyo", is predicted by GPT-2 both with and without the distractor. We use this template for all of the examples in the country capital category, swapping the antecedent "Japan" for each country provided in Mikolov et al. (2013).
Rationales for Sequential Predictions
We feed the prompts to GPT-2, which completes each analogy. To measure faithfulness, we calculate the percent of rationales that contain the true antecedent, and the percent of rationales that do not contain any words in the distractor. We only use examples where the prediction is the same both with and without the distractor. We also perform exhaustive rationale search on the objective in Equation 3. This search is highly inefficient, so we only complete it for 40 examples. To measure the approximation ratio, we divide the size of the rationale found by each method by the exhaustive rationale size.  Table 1. Language modeling faithfulness on longrange agreement with templated analogies. "Ratio" refers to the approximation ratio of each method's rationale length to the exhaustive search minimum. "Ante" refers to the percent of rationales that contain the true antecedent. "No D" refers to the percent of rationales that do not contain any tokens from the distractor.
Rationales for Sequential Predictions
has by far the least distractors in its rationales. The rationales are also universally shorter for greedy rationalization and closer to the optimal rationales, justifying our greedy assumption. To show that fine-tuning GPT-2 for compatibility is not hurting the baselines, we also perform the baseline methods on a pretrained GPT-2 without fine-tuning; see Appendix E.
Rationales for Sequential Predictions
Annotated Rationales. To test the plausibility of rationales for language models, we collect a dataset of human annotations. We base the collection on Lambada (Paperno et al., 2016), a corpus of narrative passages. Each passage included in Lambada is chosen so that humans need to use both local and global context to reliably predict the final word. By its construction it is guaranteed to have non-trivial rationales.
Rationales for Sequential Predictions
Our goal is to collect rationales that are both minimal and sufficient for humans. We run an annotation procedure with two roles: a selector and a predictor. First, the selector sees the full passage and ranks the words in order of how informative they are for predicting the final word. Next, the predictor sees one word at a time chosen by the selector, and is asked to predict the final word of the passage. The words the predictor saw before guessing the correct word form a human rationale. This rationale selection method is inspired by Rissanen Data Analysis (Rissanen, 1978;Perez et al., 2021), which uses a minimum description length metric to estimate feature importances. We rely on human annotators to estimate information gains.
Rationales for Sequential Predictions
Since it could be trivial for humans to predict the final word if it also appears in the context, we only include examples that do not repeat a word. We collect annotations for 107 examples, which we  Table 2. Language modeling plausibility on rationaleannotated Lambada.
Rationales for Sequential Predictions
It was the kind of smile that I'd seen before. The kind the boxer gave me right before he killed me in that dirty fight.
Rationales for Sequential Predictions
"I have a proposition for you" he began, pulling his hands down from under his chin and pushing out of the chair. "One that you won't be able to _____________ "Just who is going to pay for this special feed grain anyway? It must cost a bit if it's that special." "You're going to pay, obviously," replied Mitch, "since your cows will be eating it. On the other hand, Joe will be planting and irrigating the grain. He'll do all the work to make it _____________ Target word: grow Figure 5. Examples from our annotated Lambada dataset. Highlighted text denotes greedy rationales, and bolded text denotes human-annotated rationales.
Rationales for Sequential Predictions
also release publicly. We use two sets of annotators for 15% of the examples in order to compute inter-annotator agreement. On this subset, the average token-level Cohen's κ is 0.63 (Cohen, 1960), indicating substantial agreement. We compare the rationales produced by each method to the annotated rationales. Table 2 shows that the greedy rationales are most similar to the human-annotated rationales. Greedy rationalization is also the most effective at minimizing the combinatorial objective in Equation 3, as its rationales are by far the shortest. Figure 5 contains examples of rationales for this dataset.
Rationales for Sequential Predictions
It is worth noting that the top few words added by the baselines are quite relevant; after 5 tokens, the all-attention baseline has a better F1 and IOU than greedy rationalization. However, the baselines struggle to form sufficient rationales, which hurts their overall performance.
Rationales for Sequential Predictions
Distractors. To measure faithfulness, we take a transformer trained on IWSLT14 De-En (and fine-  Table 3. Translation faithfulness with distractors. "Mean crossovers" refers to the average number of crossovers per rationale, and "Crossover rate" refers to the fraction of rationales that contain at least one crossover.
Rationales for Sequential Predictions
tuned for compatibility), and generate translations for 1000 source sequences from the test set. We then create a corpus by concatenating random example pairs; for two sampled pairs of source and target sequences, (S 1 , T 1 ) and (S 2 , T 2 ), we create a new example (S 1 S 2 , T 1 T 2 ). Each token in T 1 is generated from S 1 alone, so its rationales shouldn't contain any tokens from S 2 . Similarly, T 2 is generated from S 2 alone, so its rationales shouldn't contain any tokens from S 1 or T 1 . We evaluate each rationale by counting how many times it has crossed over: a rationale for T 1 crosses over every time it contains a token in S 2 , and a rationale for T 2 crosses over every time it contains a token in S 1 or T 1 (since the model is autoregressive, T 1 's rationales can never contain tokens from T 2 ). Table 3 contains the results. Greedy rationalization has by far the fewest average number of crossovers per rationale. Although the percent of source rationales that cross over is slightly higher than the percent using gradient norms, the percentage on the target side is superior.
Rationales for Sequential Predictions
Annotated Alignments. To test plausibility, we compare the rationales to word alignments (Brown et al., 1993). Using a dataset containing 500 humanlabeled alignments for German-English translation, 5 we compute rationales for each method using the ground truth targets. We measure similarity to the labeled rationales by computing alignment error rate (AER) (Och and Ney, 2000), along with computing the IOU and F1 between sets. To separate the requirement that the rationale be sufficient from each method's global ordering of tokens, we also compare top-1 accuracies, which measure whether  Table 4. Translation plausibility with annotated alignments. The first four columns correspond to using the full source rationale found by each method; the last column "Top1" refers to the accuracy of the first source token added by each method. AER refers to alignment error rate.
Rationales for Sequential Predictions
Greedy rationale Human labeled alignment Figure 6. Greedy rationalization for machine translation. Each row depicts the source words contained in a rationale. Although each rationale includes both source and target words, here we only show source-side rationales so they can be compared to annotated alignments. the top token identified by each baseline is present in the labeled alignment set. Table 4 contains the results. The rationales learned by greedy rationalization are more similar to human-labeled alignments than those provided by gradient and attention methods. Many methods have similar top-1 accuracies -indeed, the best top-1 accuracy comes from averaging all attention layers. This reinforces the notion that although the baselines may be able to capture first-order information, they struggle to form sufficient rationales. Figure 6 contains an example of greedy rationalization applied to machine translation, along with the human-labeled alignments.
Rationales for Sequential Predictions
We proposed an optimization-based algorithm for rationalizing sequence predictions. Although exact optimization is intractable, we developed a greedy approach that efficiently finds good rationales. Moreover, we showed that models can be fine-tuned to form compatible distributions, thereby circumventing an intractable marginalization step. In experiments, we showed that the greedy algorithm is effective at optimization, and that its rationales are more faithful and plausible than those of gradient-and attention-based methods. We hope that our research, along with the release of an annotated dataset of sequence rationales, catalyzes further research into this area. 
Rationales for Sequential Predictions
We present greedy rationalization in Algorithm 1.
Rationales for Sequential Predictions
Algorithm 1: Greedy rationalization Input: Sequence y 1:t generated from p. Output: Rationale S for y t . Initialize: S = ∅ while arg max y t p(y t |y S ) = y t do k * = arg max k∈[t−1]\S p(y t |y S∪k ) S = S ∪ k * return S Mose sequence models, including transformers, use the representation of a token y t−1 to predict the next token, y t . As such, a rationale S always needs to contain y t−1 . In practice, we initialize S = {y t−1 }.
Rationales for Sequential Predictions
This method and paradigm extend easily to conditional sequence models, such as those used in machine translation. In this setting, a model uses a source sequence x 1:N to generate a target sequence y 1:T . Thus, a context for a prediction y t contains both y <t and x 1:N . The set of all possible rationales is the cross product of power sets S = 2 To perform greedy rationalization in this setting, we consider adding either a source token or a target token at each step, choosing the one that results in the largest increase in the full model's prediction.
Rationales for Sequential Predictions
Here, we show that the selection distribution q(S|x, y) that maximizes the classification rationale objective in Equation 7 is deterministic. We re-write the objective below:
Rationales for Sequential Predictions
Theorem 1. For any p(y|x S ), the q(S|x, y) that maximizes Equation 9 is a point-mass.
Rationales for Sequential Predictions
Proof. Denote by g(x, y, S) = log p(y|x S )−λ|S|:
Rationales for Sequential Predictions
Step Complexity Evaluations Total Table 5. For transformers, the asymptotic complexity of greedy rationalization matches the asymptotic complexity of forming a single prediction on the full sequence, as long as the rationale size is O(t 1/3 ) for a sequence of length t.
Rationales for Sequential Predictions
The inequality uses the fact that the expectation of a random variable is bounded by its maximum value. When q(S|x, y) is a point-mass at arg max S [g(x, y, S)], the inequality becomes tight.
Rationales for Sequential Predictions
The fact that the optimal rationale is deterministic for each example justifies using combinatorial strategies such as our objective in Equation 3.
Rationales for Sequential Predictions
In Table 5, we provide a detailed version of our complexity analysis from Section 3: For transformers, greedy rationalization can be performed at no extra asymptotic complexity if the rationale length is O(t 1/3 ) for a sequence length t.
Rationales for Sequential Predictions
We evaluate the computational efficiency of greedy rationalization in Table 6. We compare greedy rationalization to an exhaustive search, which enumerates all possible context context subsets from shortest to longest to optimize Equation 3. To show the efficiency of evaluating transformers on arbitrarily sized inputs, we also compare to a version of greedy rationalization that evaluates a transformer on the full input. To make predictions on sparse subsets, this approach masks tokens that aren't in a candidate rationale during each attention step. In contrast, the efficient version of greedy rationalization only takes as input the tokens in the candidate rationale, so there is no need for masking.
Rationales for Sequential Predictions
Exhaustive search >60 Greedy rationalization with full inputs 1.22 Greedy rationalization with sparse inputs 0.30 Table 6. Greedy rationalization is efficient, especially when evaluating transformers on sparse inputs. We report the average wall clock time in seconds for finding rationales on the templated analogies dataset of Mikolov et al. (2013). We cannot complete exhaustive search for the longer examples, so in reality the average runtime is larger than the listed one.
Rationales for Sequential Predictions
We perform these comparisons on the templated analogies dataset of Mikolov et al. (2013). We use GPT-2 Large as our sequence model (Radford et al., 2019) and perform each method on a single GPU. We compare the two greedy rationalization approaches for all of the examples for which the full model predicts the templated output. Since exhaustive search is intractable, we cannot perform it on every example due to computational constraints. Thus, we only run exhaustive search on examples where the optimal rationale has 6 or less tokens. In reality, the average runtime for exhaustive search is larger than the listed one.
Rationales for Sequential Predictions
Our experiments consist of three models and datasets: a transformer decoder (Vaswani et al., 2017) trained on a majority-class language, GPT-2 (Radford et al., 2019) fine-tuned on Open WebText (Gokaslan and Cohen, 2019), and a transformer machine translation model trained and fine-tuned with word dropout on IWSLT14 De-En (Cettolo et al., 2014).
Rationales for Sequential Predictions
For the majority-class language, we generate the dataset as described in Section 4. We include 50,000 examples in the training set, 5,000 in the validation set, and 5,000 in the test set.
Rationales for Sequential Predictions
We use a 4-layer transformer decoder with 2 attention heads per layer. We use an embedding dimension of 64, and a hidden dimension of 256 for the feedforward layers. This corresponds to 200,000 parameters. We train with 0.1 weight dropout, and optimize using Adam (Kingma and Ba, 2015) with a learning rate of 0.005 and an inverse square root learning rate scheduler. We use a warmup period of 4000 steps and an initial warmup learning rate of 10 −7 . We include a maximum of 64,000 tokens in each batch. We implement this model in Fairseq (Ott et al., 2019).
Rationales for Sequential Predictions
To approximate the compatibility objective in Equation 6, we train with varying amounts of word dropout. In practice, this amounts to masking out each token we drop out at each attention layer. We use two levels of word dropout in Figure 3; none (which corresponds to training with the standard maximum likelihood objective in Equation 1) and 0.5. We train each model on a single GPU. Each model takes less than 20,000 steps to converge, less than 90 minutes. Table 7 verifies that finetuning with word dropout does not hurt the heldout perplexity.
Rationales for Sequential Predictions
To fine-tune GPT-2, we use the pretrained GPT-2 Large model available on Hugging Face (Wolf et al., 2019). This model has 774M parameters. We don't change any of the model settings when we fine-tune. Sampling context subsets uniformly at random as stated in the objective in Equation 6 results in a distribution of subsets heavily skewed towards those containing half the words in the sequence. This is fine for the majority-class language, since each sequence contains less than 20 tokens and thus all possible context sizes will be seen during training. However, GPT-2's sequence length is 1,024. 99% of the time, sampling from the objective as stated would result in contexts with size 464-560. Notably, the probability of a context with less than 10 tokens is less than 10 −284 .
Rationales for Sequential Predictions
We make two adjustments to make sure the model is trained on both small and large subsets. With probability 0.5, we condition on the full context. With the remaining 0.5, we first randomly sample context sizes uniformly at random from 1 to the sequence length. We then sample a random context subset of this size. This guarantees that all possible sequence lengths will be seen during training.
Rationales for Sequential Predictions
Since the WebText dataset used to train GPT-2 is not publicly available, we use Open WebText (Gokaslan and Cohen, 2019), an open source reproduction effort. The corpus is in English. Rather than using the entire dataset, we take "Subset 9" and use the first 163M words. Our validation set is also from this subset and contains 160,000 words. We use a test set of 300,000 words from a different subset.
Rationales for Sequential Predictions
We fine-tune GPT-2 Large using Adam. We use a constant learning rate of 0.0001, using a single batch per training step. We stop training after 62,500 steps. This takes 15 hours on a single GPU. Table 7 shows that fine-tuning with word dropout actually improves the heldout perplexity, although we believe that the improvement is due to our test set bearing more resemblance to the fine-tuning set than to the pretraining set.
Rationales for Sequential Predictions
We use a standard transformer encoder/decoder to train a machine translation model on IWSLT14 De-En (Cettolo et al., 2014). We follow the preprocessing and model architecture recommended by Fairseq. 6 The training set has 160,239 translation pairs, the validation set has 7,283, and the test set has 6,750.
Rationales for Sequential Predictions
As for the model, both the encoder and decoder are transformers with 6 layers, 4 attention heads per layer, 512 embedding dimensions, and 1024 feedforward dimensions. This corresponds to 40M parameters. We train with 0.3 weight dropout and 0.1 label smoothing, using 4,096 tokens for each train step. We train with Adam with a learning rate of 5 × 10 −4 and use an inverse square root learning rate scheduler with 4,000 warmup steps.
Rationales for Sequential Predictions
When we fine-tune for compatibility, we again condition on the full context with probability 0.5. With the remaining probability, we drop out each source and target token independently at each attention head with probability 1 − 1/T , where T is the sequence length (so the dropout probability varies for the source and target sequence). Although we drop out different tokens at each attention head of a layer, we make sure that the same tokens are dropped out at each layer. Our word dropout procedure ensures that our objective will be trained on small contexts since rationales for machine translation are typically very sparse. We fine-tune using Adam with a constant learning rate of 10 −5 for 410,000 steps. The heldout BLEU scores of both models are equal; see Table 7 Table 8 contains the template we used for the set of experiments containing the analogies from Mikolov et al. (2013). To avoid rationales containing partial antecedents, we only include examples where both words in the analogy correspond to single word-pieces using GPT-2's tokenizer. Since it only makes sense to rationalize correct predictions, we also only include the examples where GPT-2 correctly completes the analogy. In total, this results in 175 examples. Of these, we randomly sample 50 to perform exhaustive search, which we use to compute the approximation ratio of each method. Since we cannot run exhaustive search when the minimal sufficient rationale is too large, we use the 40 that converge with rationales of length 6 or less. We use 100 steps to approximate the path integral for the integrated gradients baseline (Sundararajan et al., 2017).
Rationales for Sequential Predictions
To confirm that the baseline performances are not being hindered by fine-tuning for compatibility, we re-run the experiment for each rationalization method on the pretrained GPT-2 Large, without any fine-tuning. The results are depicted in Table 9. As expected, the baselines perform even worse when GPT-2 is not fine-tuned to form compatible distributions. We do not include comparisons to exhaustive rationales because it is computationally infeasible to run exhaustive search on incompatible models, since optimization takes much longer to converge.
Rationales for Sequential Predictions
For the distractor experiment, we randomly concatenate 500 pairs of source and target sequences generated by our fine-tuned model on the test set. We evaluate rationales by counting how many times they "cross over" and contain words from the distractor sequence. We do not penalize rationales that include special tokens like the beginning of sentence or end of sentence tokens.  Table 7. Fine-tuning for compatibility does not hurt heldout performance. The first two rows are language models and the evaluation metric is heldout perplexity; the last row is machine translation, for which the evaluation metric is BLEU.
Rationales for Sequential Predictions
Capital I really wanted to buy the computer, more than I ever wanted to buy anything before. (I was also behind on my homework, but that's another story). So I went to the store and asked if they had any computers Plural verbs I can usually sing by myself. (I was so behind on work but I tried to distract myself). Although it's so much better when someone else also sings  Table 9. The performance of each rationalization method on the templated version of the analogies dataset (Mikolov et al., 2013) when we don't fine-tune for compatibility. As expected, fine-tuning for compatibility (Table 1) improves performance across the board.
Rationales for Sequential Predictions
For the alignment experiment, we use a public corpus of annotated rationales. 7 Not every word in the dataset has an alignment, and some words have multiple alignments. Although the human annotations are on word-level alignments, our machine translation models are trained on subwords, so the rationales contain subwords in addition to full words. To make these comparable to the human annotations, we define the rationale of a full target word to contain the union of the subword rationales. Since each source word may also be a subword, we also take the union of source words in a rationale. To calculate top-1 accuracy, we define the rationale for a full word to be accurate if the rationales for any of the subwords in the rationale contain any source subwords that are in the annotated alignment.
Rationales for Sequential Predictions
The alignment dataset contains both "sure" and "possible" alignments. These are used to differentiate between different errors when calculating the alignment error rate (Och and Ney, 2000). For the other metrics, we include both kinds of alignments as part of the annotated alignments.
Rationales for Sequential Predictions
For both machine translation experiments, we use 50 steps to approximate the path integral for the integrated gradients baseline (Sundararajan et al., 2017). 7 https://www-i6.informatik.rwth-aachen. de/goldAlignment/
Rationales for Sequential Predictions
We work with volunteers to annotate Lambada (Paperno et al., 2016). Each example requires two annotators: a selector, and a predictor. A selector's goal is to choose the most important words for predicting the final word of a passage, known as the target word. Predictors will only be seeing the words chosen by a selector, and their goal is to predict the final word of the passage.
Rationales for Sequential Predictions
The selector first takes a passage and ranks 15 words. The top-ranked word always needs to be the word before the final word of the passage. They cannot select the final word of the passage. Each of their selections needs to be a complete word. They cannot select the same word twice, and they need to use all 15 spots. They know that a predictor will be predicting words, one-at-a-time, using the order they create.
Rationales for Sequential Predictions
When a selector is finished ranking the top 15 words, a predictor begins by seeing the top ranked word. They use this to predict the last word. Words are revealed one-at-a-time in the order chosen by the selector. Selectors can see how much space is between the words that have been revealed. Selectors are not told if they predicted a word correctly; the goal of the exercise is to capture the predictor's true predictions, so if they knew that previous guesses were incorrect, they may use this information to guess a new word at every step. If a predictor is not able to guess the target word at the end of the exercise, we re-assign the example to another predictor. Figure 7 contains an example given to selectors. Figure 8 contains an example given to predictors.
Rationales for Sequential Predictions
In total, we annotate 107 examples, and use all of them for the rationalization experiment. For each example, we define a human's rationale to be all the words that were revealed by the selector before the predictor first predicted the true target word or a synonym of it. The average rationale length is 6.0.
Rationales for Sequential Predictions
To compare human rationales to those found by various methods, we first tokenize the text with GPT-2's tokenizer, and convert an annotated rationale to its set of corresponding subwords. Each method's rationale is also a set of subwords. We use set-comparison metrics like intersection over union (IOU) and F1 to compare the similarity of rationales. We use 100 steps to approximate the path integral for the integrated gradients baseline (Sundararajan et al., 2017).  Target word: again I wanted to make sure you were still comfortable with the arrangements. I can always do something different."
Rationales for Sequential Predictions
You're too good to me, Max. But I'm fine. I promise. I'm going to be okay this time. I've learned from my past mistakes. I don't want to make them _____________ Target word: fire "We aren't out of danger yet," Horatius said. He headed northwest without thinking about it. It just seemed the right way to go. Chloe squirmed as she became more alert. "We have to go back. Now. We can't leave my house burning with my family in there." Horatius didn't know what to do about the _____________ Target word: ring I joined Mark, Tony, and his son back in the crowd as the event started. I wanted to catch Yegor's match before I got ready for my debut. He was wrestling first; there were only four matches on the card. Minutes after I got seated the lights dimmed. That cold music filled with horns played. It was Yegor's time to come to the _____________ 'Your services will be required for the period of three months.' I press my lips together. I was very drunk last night, but I am sure he said one month. 'Can I speak to him?' 'Of course.' He picks up the phone and speed dials his client's number. 'Mr. Barrington, Miss Bloom would like to have a word about the length of the _____________ Target word: contract Figure 9. Sample rationales from our annotated Lambada dataset. Highlighted text corresponds to greedy rationales, and bolded text corresponds to human annotated rationales. Figure 9 contains examples of rationales on our annotated Lambada dataset.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The success of neural network models in various natural language processing tasks, coupled with their opaque nature, has led to much interest in interpreting and analyzing such models. One goal of these analyses is to identify whether a model utilizes latent information in its internal representations to arrive at a prediction. This is of particular importance when diagnosing the reasons for a biased prediction. A popular class of analysis methods, often called structural analysis, aims to extract this information using probing classifiers that predict linguistic properties from representations of trained models (e.g., Adi et al., 2017;Conneau et al., 2018;Hupkes et al., 2018;Tenney et al., 2019). However, since probing classifiers only yield a correlational measure between a model's representations and an external property (Belinkov and Glass, 2019), they cannot show if the property is causally connected to the model's predictions. Moreover, Barrett et al. (2019) showed that probing classifiers may generate unfaithful interpretations and fail to generalize to unseen data.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
We introduce a methodology for interpreting neural models to address these limitations. We adapt causal mediation analysis (Pearl, 2001) for analyzing the mechanisms by which information flows from input to output through different model components. Mediation analysis relies on measuring the change in an output following a counterfactual intervention in an intermediate variable, or mediator. Through such interventions, one can measure the degree to which inputs influence outputs directly (direct effect), or indirectly through the mediator (indirect effect). In our case, the mediator can be any model components that we wish to study, such as neurons or attention heads. We propose multiple controlled interventions in these mediators, which reveal their causal role in a model's behavior.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
In a case study, we apply this framework to the analysis of gender bias in large pre-trained language models. Gender bias has surfaced as a major concern in word representations, with strong effects in both static word embeddings (Caliskan et al., 2017;Bolukbasi et al., 2016) and contextualized word representations Basta et al., 2019;Tan and Celis, 2019). Mediation analysis enables us to study how biased predictions arise from different model components. In our study, we focus on the role of individual neurons or attention heads in Transformer-based language models, in particular, several versions of GPT2 (Radford et al., 2019). In an experiment using several datasets designed to gauge a model's gender bias, we find that gender bias effects increase with larger models, which potentially absorb more bias from the underlying training data. The causal mediation analysis further reveals that gender bias is sparse, with much of the effect concentrated in a relatively small proportion of neurons and attention heads.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
In summary, this paper makes two broad contributions. First, we cast causal mediation analysis as an approach for analyzing neural NLP models, which may be applied to a variety of models and phenomena. Second, we demonstrate this methodology in the case of analyzing gender bias in pre-trained language models, revealing the internal mechanisms by which bias effects flow from input to output through various model components. The code for reproducing our results is available at https://github.com/sebastianGehrmann/CausalMediationAnalysis.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Consider a large pre-trained neural language model (LM), parameterized by θ, which predicts the probability of the next word given a prefix: p θ (x t | x 1 , . . . , x t−1 ). We will focus on LMs based on Transformers (Vaswani et al., 2017), although much of the methodology applies to other architectures as well. Let h l,i ∈ R K denote the (contextual) representation of word i in layer l of the model, with neuron activations h l,i,k (1 ≤ k ≤ K). These representations are composed using so-called multi-headed attention. Let α l,h,i,j ≥ 0 denote the attention directed from word i to word j by head h in layer l, such that j α l,h,i,j = 1. Causal mediation analysis aims to measure how a treatment effect is mediated by intermediate variables (Robins and Greenland, 1992;Pearl, 2001;Robins, 2003). Pearl (2001) described an example where a side effect of a drug may cause patients to take aspirin, and the latter has a separate effect on the disease the drug was originally prescribed for. Thus, the drug has a direct effect through its standard mechanism and an indirect effect operating via aspirin taking.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
As illustrated in Figure 1, we may consider each neuron in a neural network to be analogous to aspirin in the example above -the neuron is influenced by the input and, in turn, affects the model output; however, there also exist direct pathways from the input to the output that do not pass through the neuron. We can thus decouple model components from the rest of the model by framing them as intermediaries in the causal path from inputs to outputs. Throughout this paper, we specifically focus on the use case of gender bias in language models, as past work suggests that gender is captured in specific model components, e.g., subspaces of contextual word representations . By measuring the direct and indirect effects of targeted interventions, we can pinpoint how gender bias propagates through different parts of pre-trained LMs. While we use gender bias as a case study, the approach can be applied to other biases as well (race, ethnicity, etc.).
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Prompt u: The nurse said that Stereotypical candidate: she Anti-stereotypical candidate: he
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The example on the right illustrates a typical problem with biased LMs. Given a prompt u such as The nurse said that, a language model generates a continuation. A biased model may assign a higher likelihood to she than to he, such that p θ (she | u) > p θ (he | u). In this case, she is the stereotypical candidate, while he is the anti-stereotypical candidate, which reflects a societal bias associating nurses with women more than men. Coming back to the binary setup, the relative Here the do-operation is x = set-gender, which changes u from nurse to man in this example. The total effect measures the change in y resulting from the intervention; the direct effect measures the change in y resulting from performing the intervention while holding a mediator z fixed; the indirect effect measures the change caused by setting z to its value under the intervention, while holding u fixed.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
probabilities assigned to the candidates can be thought of as a measure of gender bias in the model:
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
In our example, y(u) = p θ (he | The nurse said that)/p θ (she | The nurse said that). If y(u) < 1, the prediction is stereotypical; if y(u) > 1, it is anti-stereotypical. A perfectly unbiased model would achieve y(u) = 1 and exhibit bias toward neither the stereotypical nor the anti-stereotypical case. This binary simplification of grammatical gender does not capture the full spectrum, as argued by Cao and Daumé III (2019), and it is not defined, for example, what probability mass a gender-neutral reference should receive 2 . While we leave extension of the framework to a continuous setup to future work, we report experimental results on the singular they compared to he.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
We then apply causal mediation analysis by performing interventions on the input text, and measuring the effect on the gender bias measure defined above (Eq. 1), which we treat as the response variable. We define the following do-operations: (a) set-gender: replace the ambiguous profession with an anti-stereotypical gender-specific word (that is, replace nurse with man, doctor with woman, etc.); (b) null: leave the sentence as is. The population of units for this analysis is a set of example sentences such as the above prompt. We define y x (u) as the value that y attains in unit u = u under the intervention do(x = x).
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The unit-level total effect (TE) of x = x on y in unit u = u is the proportional difference 3 between the amount of bias under a gendered reading and under an ambiguous reading (Figure 2a):
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
For our running example, this results in p θ (he | The man said that) p θ (she | The man said that)
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
An illustrative example of the computation of the total effect is provided in Figure 3.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The average total effect of x = x on y is calculated by taking the expectation over the population u:
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
We then analyze the causal role of specific mediators, or intermediary variables, which lie between x and y. The mediator, denoted as z, might be a particular neuron, a full layer, an attention head, or a certain attention weight. Following Pearl's definitions, we measure the direct and indirect effects of intervening in the model relative to z (Pearl, 2001).  Figure 3: An example calculation of the total effect with the prompt u = The nurse said that and the control variable x = set-gender. Before the intervention, the model assigns a much higher probability to [she], the stereotypical example, than to [he]. By changing nurse to man, we compute the proportional probability of a definitionally gendered example. The total effect measures the effect of this intervention.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The natural direct effect (NDE) measures how much an intervention x changes an outcome variable y directly, without passing through a hypothesized mediator z. It is computed by applying the intervention x but holding z fixed to its original value. For the present use case, we define the NDE of x = x on y given mediator z = z to be the change in the amount of bias when genderizing all units u, e.g., changing nurse to man, while holding z for each unit to its original value. This measures the direct effect on gender bias that does not pass through the mediator z (illustrated in Figure 2b):
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The natural indirect effect (NIE) measures how much the intervention x changes y indirectly, through z. It is computed by setting z to its value under the intervention x, while keeping everything else to its original value. Thus the indirect effect captures the influence of a mediator on the outcome variable. For the present use case, we define the NIE as the change in amount of bias when keeping unit u as is, but setting z to the value it would attain under a genderized reading. This measures the indirect effect flowing from x to y through z ( Figure 2c):
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
NIE(set-gender, null; y) = E u [y null,zset-gender(u) (u)/y null (u) − 1].
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
This framework allows evaluating the causal contribution of different mediators z to gender bias. Through the distinction between direct and indirect effect, we can measure how much of the total effect of gender edits on gender bias flows through a specific component (indirect effect) or elsewhere in the model (direct effect). We experiment with mediators at the neuron level and the attention level.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
To study the role of individual neurons in mediating gender bias, we assign z to each neuron h l,·,k in the LM. The dataset we use consists of a list of templates that are instantiated by profession terms, resulting in examples such as The nurse said that. For each example, we define the set-gender operation to move in the anti-stereotypical direction, changing female-stereotypical professions like nurse to man and male-stereotypical professions like doctor to woman. Section 3 provides more information on the dataset. We additionally investigate the effect of a gender-neutral intervention, for which we pick person as target of the set-gender change and we measure the probability of the continuation they. Note that, unfortunately, all examples can be seen as biased against genderneutrality since the models have had limited exposure to the singular they. Moreover, this case suffers from the additional confounder that the model could assign probability to the plural they if it does not refer to the profession.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
In the experiments, we investigate the effect of intervening on each neuron independently, as well as on multiple neurons concurrently. That is, the mediator z may be a set of neurons. In all cases, the mediator is in the representation corresponding to the profession word, such as nurse in the example.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
For studying attention behavior, we focus on the attention weights, which define relationships between words. The mediators z, in this case, are the attention heads α l,h , each of which defines a distinct attention mechanism. Prompt u: The nurse examined the farmer for injuries because she Stereotypical candidate: was caring Anti-stereotypical candidate: was screaming
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
To study their role, we align our intervention approach with two resources for assessing gender bias in pronoun resolution: Winobias (Zhao et al., 2018a) and Winogender (Rudinger et al., 2018). Both datasets consist of Winograd-schema-style examples that aim to assess gender bias in coreference resolution systems. We reformulate the examples to study bias in LMs, as shown in the example on the right, taken from Winobias. According to the stereotypical reading, the pronoun she refers to the nurse, implying the continuation was caring. The anti-stereotypical reading links she to the farmer, this time implying the continuation was screaming. The bias measure is y(u) = p θ (was screaming | u)/p θ (was caring | u). 4 In this case, we define the swap-gender operation, which changes she to he. The total effect is
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
In the experiments, we study the effect of the attention from the last word (she or he) to the rest of the sentence. 5 Intuitively, in the above example, if the word she attends more to nurse than to farmer, then the more likely continuation might be was caring. We compute the NDE and NIE for each head individually by intervening on the attention weights α l,h,·,· . We also evaluate the joint effects when intervening on multiple attention heads concurrently. The population-level TE and the NDE and NIE are defined analogously as above.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Models As an example large pre-trained LM, we use GPT2 ( For attention intervention experiments, we use examples from Winobias Dev/Test (Zhao et al., 2018a) and Winogender (Rudinger et al., 2018), totaling 160/130 and 44 examples that fit our formulation, respectively. We experiment with the full datasets and filtering by total effect. Both datasets include statistics from the U.S. Bureau of Labor Statistics to assess the gender stereotypicality of the referenced occupations. Appendix A.2 provides additional details about the datasets and preprocessing methods. the binary male→female or female→male interventions. We report separately the results of male/female→neutral interventions, which due to their potentially confounded nature cannot be grouped with the rest of the results. Table 1 shows the total effects of gender bias in the different GPT2 models, on three datasets, as well as the effects with a randomly initialized GPT2-small model. Random model effects are much smaller, indicating that it is the training process that causes gender bias.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Larger models are more sensitive to gender bias In the Winograd-style datasets, the total effect mostly increases with model size, saturating at the large and xl models. In the professions dataset, model size is not well correlated with total effect, but GPT2-xl has a much larger effect. Since larger models can more accurately emulate the training corpus, it makes sense that they would more strongly integrate its biases.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
It is difficult to compare effect magnitudes in the three datasets because of their different nature. The professions dataset yields much stronger effects than the Winograd-style datasets. This may be attributed to the more explicit source of bias, the word representations, as compared to intricate coreference relations in the Winograd-style datasets.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Some effects are correlated with external gender statistics In the professions dataset, we found moderate positive correlations between external gender bias 7 and the log-total effect, ranging from 0.35 to 0.45 over different models, indicating that the model captures the expected biases. It further shows that the effect is amplified by the model for words that are perceived as more biased. In the Winograd-style datasets, we found relatively low correlations between the log-total effect and the log-ratio of the two occupations' stereotypicality, ranging from 0.17 to 0.26. This low correlation may be due to a smaller size than the professions dataset or the more complex Winograd-style relations.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The gender-neutral case leads to more consistent effects In the neutral case, the baseline probability p(they|u) is much more consistent, but low, across all professions. Consider the template "The X said that" -in this case, under GPT2-distil "they" varies in probability from 0.2% to 4.2% while "he" has a much wider range from 1.1% to 31.8%. Consequently, the total effect for neutral interventions is much more consistent across models and templates. GPT2-distill, GPT2-small, and GPT2-medium have total effects of 8.3, 7.5, and 9.6 respectively, all with standard deviations < 10, in the professions dataset. We hypothesize that this can mostly be attributed to very low probability for the singular "they" and a consistent baseline probability where "they" is part of a referential statement toward a group of individuals, for example in "The accountant said that they [the people] need to pay taxes".
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Where in the model are gender bias effects captured? Are the effects mediated by only a few model components or distributed across the model? Here we answer these questions by measuring the indirect effect flowing through different mediators.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Attention Figure 4a shows the indirect effects for each head in GPT2-small on Winobias. The heatmap shows interventions on each head individually. A small number of heads, concentrated in the middle layers of the model, have much higher indirect effects than others. The bar chart shows indirect effects when intervening on all heads in a single layer concurrently. Consistent with the head-level heatmap, the effects are concentrated in the middle layers. We found this sparsity consistent in all model variants and datasets we examined. We did not find similar behavior in a  : Attention in GPT2-small on a Winobias example, directed from either she or he. Head 5-10 attends directly to the bold stereotypical candidate, head 5-8 attends to the words following it, and head 4-6 attends to the underlined antistereotypical candidate. Attention to the first token may be null attention (Vig and Belinkov, 2019). Appendix C.2 shows more examples.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
randomly initialized model, indicating that these patterns do not occur by chance. Appendix C.1 provides additional visualizations of indirect effects as well as direct effects.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The indirect and direct effects of the top attention heads are summarized in Figure 5. The total effect roughly equals the sum of the direct and indirect effects. 8 Qualitative analysis suggests that these top heads take on specialized roles with respect to gender bias, as illustrated in Figure 6. The figure demonstrates that attention heads capture different coreference aspects: one head aligns with the stereotypical coreference candidate, another head attends to the tokens following that candidate, while a third attends to the anti-stereotypical candidate. Vig (2019)  To determine how many heads are required to achieve the full effect of intervening on all heads, we also intervene on groups of heads. While the computational complexity of selecting a single head scales linearly with the number of total heads, selecting a group of heads scales polynomially and becomes computationally intractable. To efficiently select a subset of k heads given n total heads, we use two methods: a GREEDY approach, which iteratively selects the head with the maximal marginal contribution to the indirect effect and requires O(nk) evaluations, and a TOP-K approach,  Neurons Figure 7a shows the indirect effects from the top 5% of neurons from each layer in different models. The word embeddings (layer 0) and the first hidden layer have the strongest effects. This stands in contrast to the attention intervention results, where middle layers had much larger effects. However, we still observe a small increase in effect within the intermediate layers across all models except for the randomized one. Interestingly, we do not observe the same concentration for neutral intervention.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
As can be seen in Figure 8, where, for simplicity, we focus on GPT2-medium, the effects are distributed across all layers, but similarly increasing a bit toward the later middle layers.   information learned by the probing classifier from that learned by the underlying model (Hewitt and Liang, 2019). Behavioral approaches, on the other hand, assess how well a model captures different linguistic phenomena by evaluating the model's performance on curated examples (e.g., Sennrich, 2017;Isabelle et al., 2017;Naik et al., 2018). These methods directly evaluate a model's prediction but fail to provide insight into its internal structure. Another approach identifies important input features that contribute to a model's prediction via saliency methods (Li et al., 2016;Arras et al., 2017;Murdoch et al., 2018), which also typically ignore the model's internal structure, although one could compute them with respect to internal representations.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Our causal mediation analysis approach bridges the gap between these two lines of work, providing an analysis that is both structural and behavioral. Mediation analysis is an unexplored formulation in the context of interpreting deep NLP models. In recent work, Zhao and Hastie (2019) used mediation analysis for interpreting black-box models. However, their analysis was limited to simple datasets and models, while we focus on deep language models. Furthermore, they only considered total effects and (controlled) direct effects, while we measure (natural) direct and indirect effects, which is crucial for studying the role of internal model components.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Neural networks learn to replicate historical, societal biases from training data in various tasks such as natural language inference (Rudinger et al., 2017), coreference resolution (Cao and Daumé III, 2019), and sentiment analysis (Kiritchenko and Mohammad, 2018). This conflicts with the principle of counterfactual fairness, which states that the model predictions should not be influenced by changes to a sensitive attribute such as gender (Kusner et al., 2017); for instance, a fair and unbiased model should equally associate gendered pronouns with professions. However, biased models make this association proportionally to the distribution of gender in the training data (Caliskan et al., 2017). While efforts have been made to reduce bias, this remains a significant ethical challenge.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
A common strategy to mitigate biases is to change the training data (e.g., Lu  Our work contributes a novel perspective to the literature on gender bias in neural NLP model by characterizing the role of mediators in biased predictions via performing interventions.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
This paper introduced a framework for interpreting neural NLP models based on causal mediation analysis. An application of this framework yields several insights regarding the mechanisms by which gender bias is mediated in Transformer LMs. We find that larger models have a greater capacity to absorb gender bias, though this bias manifests in a relatively small proportion of neurons and attention heads. Qualitative analysis suggests that model components may take on specialized roles in propagating gender bias.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
This framework can be extended in multiple ways, for example to work with different model architectures and to analyze different and potentially continuous or multi-class biases (Cao and Daumé III, 2019). The results could also be applied to control model outputs to generate text with fair representation (Giulianelli et al., 2018;Dathathri et al., 2020). The causality literature offers many avenues for continuing this line of work, including mediation analysis with non-linear models, and alternative effect decompositions (Imai et al., 2010a,b;VanderWeele and Vansteelandt, 2009). A promising direction is to focus on path-specific effects (Avin et al., 2005), to identify the exact mechanisms through which biases arise. Characterizing specific paths from model input to output might also be useful during training by disincentivizing the creation of paths leading to bias.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
This work focuses on the identification and analysis of biases that large language models acquire during training. Following the reasoning of Rawls (1958) among others, it is impermissible to use models that treat persons, groups, or institutions differently based on their attributes. Yet, language models are widely applied in real world settings. To remedy the effect of the implicit discrimination that this may cause, it is imperative to develop unbiased models. Understanding the causal mechanisms within neural networks is critical to developing trustworthy and provably fair models. Our method presents a first step toward the active debiasing of such models, as discussed in Section 6. Moreover, since model biases mirror societal biases, as shown by Caliskan et al. (2017) and confirmed in Section 4.1, our method may be of interest to those studying these biases in large corpora.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
However, while our case study presents a best effort to cover different cases and linguistic phenomena, it is not possible to fully cover all cases of gender bias within a language using only a limited set of constructed templates. Importantly, the main focus of our study uses a limited binary setup, which does not easily lend itself to applications on grammatical gender. We tried to avoid implicit discrimination of individuals who do not identify as either male or female by conducting experiments on a gender-neutral pronoun, but more work needs to be done on generating inclusive referring expressions that cover all possible pronouns. Moreover, since the model confuses the singular for the plural they, it will require additional disambiguation efforts to apply our methodology in this case. Finally, this study focuses only on the English language. The conclusions drawn from our results may not generalize to other languages or linguistic phenomena. In this case, our findings may lead researchers down the wrong path.  
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
A.1 Professions Dataset Figure 9 shows the 17 base templates used for the neuron interventions. To validate that each template would capture gender bias, we instantiate each with an occupation of woman and man and verify that the conditional probabilities of she and he align with gender. Given woman as the occupation word, the probability ratio p(she)/p(he) ranges from 2.5 to 45.1 across templates (µ = 17.2, σ = 13.1).
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Given man, the value p(he)/p(she) ranges from 3.0 to 55.4 (µ = 21.9, σ = 16.2). Thus the relative probabilities align with gender, though they vary greatly in magnitude.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
For each of the templates, we used the following professions. Words in italics are definitional and were thus excluded from the total effect calculation: female: actress, advocate, aide, artist, baker, clerk, counselor, dancer, educator, instructor, maid, nun, nurse, observer, performer, photographer, planner, poet, protester, psychiatrist, secretary, singer, substitute, teacher, teenager, therapist, treasurer, tutor, waitress neutral: acquaintance, character, citizen, correspondent, employee, musician, novelist, psychologist, student, writer male: accountant, actor, administrator, adventurer, ambassador, analyst, architect, assassin, astronaut, astronomer, athlete, attorney, author, banker, bartender, biologist, bishop, boss, boxer, broadcaster, broker, businessman, butcher, campaigner, captain, chancellor, chef, chemist, cleric, coach, collector, colonel, columnist, comedian, comic, commander, commentator, commissioner, composer, conductor, congressman, consultant, cop, critic, curator, dad, dean, dentist, deputy, detective, diplomat, director, doctor, drummer, economist, editor, entrepreneur, envoy, farmer, filmmaker, firefighter, fisherman, footballer, goalkeeper, guitarist, historian, inspector, inventor, investigator, journalist, judge, landlord, lawmaker, lawyer, lecturer, legislator, lieutenant, magician, magistrate, manager, mathematician, mechanic, medic, midfielder, minister, missionary, monk, narrator, negotiator, officer, painter, pastor, philosopher, physician, physicist, policeman, politician, preacher, president, priest, principal, prisoner, professor, programmer, promoter, prosecutor, protagonist, rabbi, ranger, researcher, sailor, saint, salesman, scholar, scientist, senator, sergeant, servant, soldier, solicitor, strategist, superintendent, surgeon, technician, trader, trooper, waiter, warrior, worker, wrestler 
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
For both Winobias and Winogender datasets, we exclude templates in which the shared prompt does not end in a pronoun. 10 For Winobias, we only consider Type 1 examples, which follow the format of a shared prompt and two alternate continuations. We also experiment with filtering by total effect,  Both datasets include statistics from the U.S. Bureau of Labor Statistics (BLS) to assess the gender stereotypicality of the referenced occupations. Winogender additionally includes gender estimates from text (Bergsma and Lin, 2006), which we also include in our analysis. Whereas each Winobias example includes two occupations of opposite stereotypicality, each Winogender example includes one occupation and a participant, for which no gender statistics are provided. For consistency with the Winobias analysis, we make the simplifying assumption that the gender stereotypicality of the participant is the opposite of that of the occupation.  Table 4 provides the total effects on the professions dataset when separated to stereotypically female and male professions, where stereotypicality is defined by the profession statistics provided by Bolukbasi et al. (2016). Notably, the effects are much larger in the female case. This may be explained by stereotypicaly-female professions being of higher stereotypicality than stereotypicallymale professions, reflecting a societal bias viewing women's professions as more narrowed.   Figure 4a, the attention heads with the largest indirect effects lie in the middle layers of each model. Figure 11 shows the indirect effects for a model with randomized weights. Figures 12 and 13 visualize the indirect effects for other dataset variations for the GPT2-small model from Figure 4a. The attention heads with largest indirect effect have significant overlap across the dataset variations. 
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
We wish to select a subset of attention heads or neurons that perform well together to better understand the sparsity of attention heads and neurons and their impact on gender bias in Transformer models.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
The problem of subset selection (selecting k elements from n) is an NP-hard combinatorial optimization problem. To construct a meaningful solution set, we employ several algorithms for subset selection from submodular maximization. We note that while our objective functions are not strictly submodular as they do not satisfy the diminishing returns property, our objectives exhibit submodularlike properties and numerous algorithms have been proposed to efficiently maximize submodular and variants of submodular functions.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
For monotone submodular functions, it is known that a greedy algorithm that iteratively selects the element with the maximal marginal contribution to its current solution obtains a 1−1/e approximation for maximization under a cardinality constraint (Nemhauser and Wolsey, 1978) and that this bound is optimal. For non-monotone submodular functions, there is the randomized greedy algorithm which emits a 1/e approximation to the optimal solution (Buchbinder et al., 2014).
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
To select subsets of attention heads, we compare TOP-K (selecting k elements with the largest individual values) and GREEDY. Even though randomized greedy has stronger theoretical guarantees because our objective is clearly non-monotonic, we favor the deterministic algorithm for increased interpretability. Figure 17 shows results for head selection across different models on Winogender and Winobias. Sparsity is consistent across all experiments where only a small proportion of heads are sufficient to achieve the full model effect of intervening at all heads. On Winogender, only 4/4/5/4% of heads are needed to saturate, while on Winobias, only 6/7/8/6% of heads are needed in GPT2-distil/small/medium/large.
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
To select subsets of neurons, we use TOP-K to compute NIE of sets of neurons because sequential greedy is too computationally intensive to run. Alternative methods using adaptive sampling techniques have been proposed to speed-up GREEDY for submodular functions under cardinality constraints (Ene and Nguyen, 2019;Fahrbach et al., 2019b;Balkanski and Singer, 2018a,b). For nonmonotone or non-submodular functions, there are parallelized algorithms that use similar techniques to select sets (Balkanski et al., 2018;Qian and Singer, 2019;Fahrbach et al., 2019a). These methods provide an alternative approach to TOP-K for selecting subsets of neurons and can be explored in future work. 
Rationalizing Neural Predictions
Many recent advances in NLP problems have come from formulating and training expressive and elaborate neural models. This includes models for sentiment classification, parsing, and machine translation among many others. The gains in accuracy have, however, come at the cost of interpretability since complex neural models offer little transparency concerning their inner workings. In many applications, such as medicine, predictions are used to drive critical decisions, including treatment options. It is necessary in such cases to be able to verify and under-the beer was n't what i expected, and i'm not sure it's "true to style", but i thought it was delicious. a very pleasant ruby red-amber color with a rela9vely brilliant finish, but a limited amount of carbona9on, from the look of it. aroma is what i think an amber ale should be -a nice blend of caramel and happiness bound together.
Rationalizing Neural Predictions
Look: 5 stars Smell: 4 stars Figure 1: An example of a review with ranking in two categories. The rationale for Look prediction is shown in bold.
Rationalizing Neural Predictions
stand the underlying basis for the decisions. Ideally, complex neural models would not only yield improved performance but would also offer interpretable justifications -rationales -for their predictions.
Rationalizing Neural Predictions
In this paper, we propose a novel approach to incorporating rationale generation as an integral part of the overall learning problem. We limit ourselves to extractive (as opposed to abstractive) rationales. From this perspective, our rationales are simply subsets of the words from the input text that satisfy two key properties. First, the selected words represent short and coherent pieces of text (e.g., phrases) and, second, the selected words must alone suffice for prediction as a substitute of the original text. More concretely, consider the task of multi-aspect sentiment analysis. Figure 1 illustrates a product review along with user rating in terms of two categories or aspects. If the model in this case predicts five star rating for color, it should also identify the phrase "a very pleasant ruby red-amber color" as the rationale underlying this decision.
Rationalizing Neural Predictions
In most practical applications, rationale genera-tion must be learned entirely in an unsupervised manner. We therefore assume that our model with rationales is trained on the same data as the original neural models, without access to additional rationale annotations. In other words, target rationales are never provided during training; the intermediate step of rationale generation is guided only by the two desiderata discussed above. Our model is composed of two modular components that we call the generator and the encoder. Our generator specifies a distribution over possible rationales (extracted text) and the encoder maps any such text to task specific target values. They are trained jointly to minimize a cost function that favors short, concise rationales while enforcing that the rationales alone suffice for accurate prediction.
Rationalizing Neural Predictions
The notion of what counts as a rationale may be ambiguous in some contexts and the task of selecting rationales may therefore be challenging to evaluate. We focus on two domains where ambiguity is minimal (or can be minimized). The first scenario concerns with multi-aspect sentiment analysis exemplified by the beer review corpus (McAuley et al., 2012). A smaller test set in this corpus identifies, for each aspect, the sentence(s) that relate to this aspect. We can therefore directly evaluate our predictions on the sentence level with the caveat that our model makes selections on a finer level, in terms of words, not complete sentences. The second scenario concerns with the problem of retrieving similar questions. The extracted rationales should capture the main purpose of the questions. We can therefore evaluate the quality of rationales as a compressed proxy for the full text in terms of retrieval performance. Our model achieves high performance on both tasks. For instance, on the sentiment prediction task, our model achieves extraction accuracy of 96%, as compared to 38% and 81% obtained by the bigram SVM and a neural attention baseline.
Rationalizing Neural Predictions
We formalize here the task of extractive rationale generation and illustrate it in the context of neural models. To this end, consider a typical NLP task where we are provided with a sequence of words as input, namely x = {x 1 , · · · , x l }, where each x t ∈ R d denotes the vector representation of the ith word. The learning problem is to map the input sequence x to a target vector in R m . For example, in multi-aspect sentiment analysis each coordinate of the target vector represents the response or rating pertaining to the associated aspect. In text retrieval, on the other hand, the target vectors are used to induce similarity assessments between input sequences. Broadly speaking, we can solve the associated learning problem by estimating a complex parameterized mapping enc(x) from input sequences to target vectors. We call this mapping an encoder. The training signal for these vectors is obtained either directly (e.g., multi-sentiment analysis) or via similarities (e.g., text retrieval). The challenge is that a complex neural encoder enc(x) reveals little about its internal workings and thus offers little in the way of justification for why a particular prediction was made.
Rationalizing Neural Predictions
In extractive rationale generation, our goal is to select a subset of the input sequence as a rationale. In order for the subset to qualify as a rationale it should satisfy two criteria: 1) the selected words should be interpretable and 2) they ought to suffice to reach nearly the same prediction (target vector) as the original input. In other words, a rationale must be short and sufficient. We will assume that a short selection is interpretable and focus on optimizing sufficiency under cardinality constraints.
Rationalizing Neural Predictions
We encapsulate the selection of words as a rationale generator which is another parameterized mapping gen(x) from input sequences to shorter sequences of words. Thus gen(x) must include only a few words and enc(gen(x)) should result in nearly the same target vector as the original input passed through the encoder or enc(x). We can think of the generator as a tagging model where each word in the input receives a binary tag pertaining to whether it is selected to be included in the rationale. In our case, the generator is probabilistic and specifies a distribution over possible selections.
Rationalizing Neural Predictions
The rationale generation task is entirely unsupervised in the sense that we assume no explicit annotations about which words should be included in the rationale. Put another way, the rationale is introduced as a latent variable, a constraint that guides how to interpret the input sequence. The encoder and generator are trained jointly, in an end-to-end fashion so as to function well together.
Rationalizing Neural Predictions
We use multi-aspect sentiment prediction as a guiding example to instantiate the two key componentsthe encoder and the generator. The framework itself generalizes to other tasks.
Rationalizing Neural Predictions
Encoder enc(·): Given a training instance (x, y) where x = {x t } l t=1 is the input text sequence of length l and y ∈ [0, 1] m is the target m-dimensional sentiment vector, the neural encoder predictsỹ = enc(x). If trained on its own, the encoder would aim to minimize the discrepancy between the predicted sentiment vectorỹ and the gold target vector y. We will use the squared error (i.e. L 2 distance) as the sentiment loss function,
Rationalizing Neural Predictions
The encoder could be realized in many ways such as a recurrent neural network. For example, let h t = f e (x t , h t−1 ) denote a parameterized recurrent unit mapping input word x t and previous state h t−1 to next state h t . The target vector is then generated on the basis of the final state reached by the recurrent unit after processing all the words in the input sequence. Specifically,
Rationalizing Neural Predictions
Generator gen(·): The rationale generator extracts a subset of text from the original input x to function as an interpretable summary. Thus the rationale for a given sequence x can be equivalently defined in terms of binary variables {z 1 , · · · , z l } where each z t ∈ 0, 1 indicates whether word x t is selected or not. From here on, we will use z to specify the binary selections and thus (z, x) is the actual rationale generated (selections, input). We will use generator gen(x) as synonymous with a probability distribution over binary selections, i.e., z ∼ gen(x) ≡ p(z|x) where the length of z varies with the input x.
Rationalizing Neural Predictions
In a simple generator, the probability that the t th word is selected can be assumed to be conditionally independent from other selections given the input x. That is, the joint probability p(z|x) factors according to
Rationalizing Neural Predictions
The component distributions p(z t |x) can be modeled using a shared bi-directional recurrent neural network. Specifically, let − → f () and ← − f () be the forward and backward recurrent unit, respectively, then
Rationalizing Neural Predictions
Independent but context dependent selection of words is often sufficient. However, the model is unable to select phrases or refrain from selecting the same word again if already chosen. To this end, we also introduce a dependent selection of words,
Rationalizing Neural Predictions
which can be also expressed as a recurrent neural network. To this end, we introduce another hidden state s t whose role is to couple the selections. For example,
Rationalizing Neural Predictions
Joint objective: A rationale in our definition corresponds to the selected words, i.e., {x k |z k = 1}. We will use (z, x) as the shorthand for this rationale and, thus, enc(z, x) refers to the target vector obtained by applying the encoder to the rationale as the input. Our goal here is to formalize how the rationale can be made short and meaningful yet function well in conjunction with the encoder. Our generator and encoder are learned jointly to interact well but they are treated as independent units for modularity.
Rationalizing Neural Predictions
The generator is guided in two ways during learning. First, the rationale that it produces must suffice as a replacement for the input text. In other words, the target vector (sentiment) arising from the rationale should be close to the gold sentiment. The corresponding loss function is given by
Rationalizing Neural Predictions
Note that the loss function depends directly (parametrically) on the encoder but only indirectly on the generator via the sampled selection.
Rationalizing Neural Predictions
Second, we must guide the generator to realize short and coherent rationales. It should select only a few words and those selections should form phrases (consecutive words) rather than represent isolated, disconnected words. We therefore introduce an additional regularizer over the selections
Rationalizing Neural Predictions
where the first term penalizes the number of selections while the second one discourages transitions (encourages continuity of selections). Note that this regularizer also depends on the generator only indirectly via the selected rationale. This is because it is easier to assess the rationale once produced rather than directly guide how it is obtained.
Rationalizing Neural Predictions
Our final cost function is the combination of the two, cost(z, x, y) = L(z, x, y) + Ω(z). Since the selections are not provided during training, we minimize the expected cost:
Rationalizing Neural Predictions
where θ e and θ g denote the set of parameters of the encoder and generator, respectively, and D is the collection of training instances. Our joint objective encourages the generator to compress the input text into coherent summaries that work well with the associated encoder it is trained with.
Rationalizing Neural Predictions
Minimizing the expected cost is challenging since it involves summing over all the possible choices of rationales z. This summation could potentially be made feasible with additional restrictive assumptions about the generator and encoder. However, we assume only that it is possible to efficiently sample from the generator.
Rationalizing Neural Predictions
Doubly stochastic gradient We now derive a sampled approximation to the gradient of the expected cost objective. This sampled approximation is obtained separately for each input text x so as to work well with an overall stochastic gradient method. Consider therefore a training pair (x, y). For the parameters of the generator θ g ,
Rationalizing Neural Predictions
The last term is the expected gradient where the expectation is taken with respect to the generator distribution over rationales z. Therefore, we can simply sample a few rationales z from the generator gen(x) and use the resulting average gradient in an overall stochastic gradient method. A sampled approximation to the gradient with respect to the encoder parameters θ e can be derived similarly,
Rationalizing Neural Predictions
Choice of recurrent unit We employ recurrent convolution (RCNN), a refinement of local-ngram based convolution. RCNN attempts to learn n-gram features that are not necessarily consecutive, and average features in a dynamic (recurrent) fashion. Specifically, for bigrams (filter width n = 2) RCNN computes h t = f (x t , h t−1 ) as follows Number of reviews 1580k Avg length of review 144.9 Avg correlation between aspects 63.5% Max correlation between two aspects 79.1% Number of annotated reviews 994 
Rationalizing Neural Predictions
RCNN has been shown to work remarkably in classification and retrieval applications (Lei et al., 2015;Lei et al., 2016) compared to other alternatives such CNNs and LSTMs. We use it for all the recurrent units introduced in our model.
Rationalizing Neural Predictions
We evaluate the proposed joint model on two NLP applications: (1) multi-aspect sentiment analysis on product reviews and (2) similar text retrieval on AskUbuntu question answering forum.
Rationalizing Neural Predictions
Dataset We use the BeerAdvocate 2 review dataset used in prior work (McAuley et al., 2012). 3 This dataset contains 1.5 million reviews written by the website users. The reviews are naturally multiaspect -each of them contains multiple sentences describing the overall impression or one particular aspect of a beer, including appearance, smell (aroma), palate and the taste. In addition to the written text, the reviewer provides the ratings (on a scale of 0 to 5 stars) for each aspect as well as an overall rating. The ratings can be fractional (e.g. 3.5 stars), so we normalize the scores to [0, 1] and use them as the (only) supervision for regression. McAuley et al. (2012) also provided sentencelevel annotations on around 1,000 reviews. Each sentence is annotated with one (or multiple) aspect label, indicating what aspect this sentence covers.   We use this set as our test set to evaluate the precision of words in the extracted rationales. Table 1 shows several statistics of the beer review dataset. The sentiment correlation between any pair of aspects (and the overall score) is quite high, getting 63.5% on average and a maximum of 79.1% (between the taste and overall score). If directly training the model on this set, the model can be confused due to such strong correlation. We therefore perform a preprocessing step, picking "less correlated" examples from the dataset. 4 This gives us a de-correlated subset for each aspect, each containing about 80k to 90k reviews. We use 10k as the development set. We focus on three aspects since the fourth aspect taste still gets > 50% correlation with the overall sentiment.
Rationalizing Neural Predictions
Before training the joint model, it is worth assessing the neural encoder separately to check how accurately the neural network predicts the sentiment. To this end, we compare neural encoders with bigram SVM model, training medium and large SVM models using 260k and all 4 Specifically, for each aspect we train a simple linear regression model to predict the rating of this aspect given the ratings of the other four aspects. We then keep picking reviews with largest prediction error until the sentiment correlation in the selected subset increases dramatically. nales (x-axis). 220k training data is used. 1580k reviews respectively. As shown in Table 3, the recurrent neural network models outperform the SVM model for sentiment prediction and also require less training data to achieve the performance. The LSTM and RCNN units obtain similar test error, getting 0.0094 and 0.0087 mean squared error respectively. The RCNN unit performs slightly better and uses less parameters. Based on the results, we choose the RCNN encoder network with 2 stacking layers and 200 hidden states.
Rationalizing Neural Predictions
To train the joint model, we also use RCNN unit with 200 states as the forward and backward recurrent unit for the generator gen(). The dependent generator has one additional recurrent layer. For this layer we use 30 states so the dependent version still has a number of parameters comparable to the independent version. The two versions of the generator have 358k and 323k parameters respectively. Figure 2 shows the performance of our joint dependent model when trained to predict the sentiment of all aspects. We vary the regularization λ 1 and λ 2 to show various runs that extract different amount of text as rationales. Our joint model gets performance close to the best encoder run (with full text) when few words are extracted. a beer that is not sold in my neck of the woods , but managed to get while on a roadtrip . poured into an imperial pint glass with a generous head that sustained life throughout . nothing out of the ordinary here , but a good brew s9ll . body was kind of heavy , but not thick . the hop smell was excellent and en9cing . very drinkable very dark beer . pours a nice finger and a half of creamy foam and stays throughout the beer . smells of coffee and roasted malt . has a major coffee-like taste with hints of chocolate . if you like black coffee , you will love this porter . creamy smooth mouthfeel and definitely gets smoother on the palate once it warms . it 's an ok porter but i feel there are much beAer one 's out there .
Rationalizing Neural Predictions
poured into a sniBer . produces a small coffee head that reduces quickly . black as night . preAy typical imp . roasted malts hit on the nose . a liAle sweet chocolate follows . big toasty character on the taste . in between i 'm geDng plenty of dark chocolate and some biAer espresso . it finishes with hop biAerness . nice smooth mouthfeel with perfect carbona9on for the style . overall a nice stout i would love to have again , maybe with some age on it .
Rationalizing Neural Predictions
i really did not like this . it just seemed extremely watery . i dont ' think this had any carbona9on whatsoever . maybe it was flat , who knows ? but even if i got a bad brew i do n't see how this would possibly be something i 'd get 9me and 9me again . i could taste the hops towards the middle , but the beer got preAy nasty towards the boAom . i would never drink this again , unless it was free . i 'm kind of upset i bought this . a : poured a nice dark brown with a tan colored head about half an inch thick , nice red/garnet accents when held to the light . liAle clumps of lacing all around the glass , not too shabby . not terribly impressive though s : smells like a more guinness-y guinness really , there are some roasted malts there , signature guinness smells , less burnt though , a liAle bit of chocolate … … m : rela9vely thick , it is n't an export stout or imperial stout , but s9ll is preAy heBy in the mouth , very smooth , not much carbona9on . not too shabby d : not quite as drinkable as the draught , but s9ll not too bad . i could easily see drinking a few of these . Rationale Selection To evaluate the supporting rationales for each aspect, we train the joint encodergenerator model on each de-correlated subset. We set the cardinality regularization λ 1 between values {2e − 4, 3e − 4, 4e − 4} so the extracted rationale texts are neither too long nor too short. For simplicity, we set λ 2 = 2λ 1 to encourage local coherency of the extraction.
Rationalizing Neural Predictions
For comparison we use the bigram SVM model and implement an attention-based neural network model. The SVM model successively extracts unigram or bigram (from the test reviews) with the highest feature. The attention-based model learns a normalized attention vector of the input tokens (using similarly the forward and backward RNNs), then the model averages over the encoder states accordingly to the attention, and feed the averaged vector to the output layer. Similar to the SVM model, the attention-based model can selects words based on their attention weights. The smell (aroma) aspect is the target aspect. Table 2 presents the precision of the extracted rationales calculated based on sentence-level aspect annotations. The λ 1 regularization hyper-parameter is tuned so the two versions of our model extract similar number of words as rationales. The SVM and attention-based model are constrained similarly for comparison. Figure 4 further shows the precision when different amounts of text are extracted. Again, for our model this corresponds to changing the λ 1 regularization. As shown in the table and the figure, our encoder-generator networks extract text pieces describing the target aspect with high precision, ranging from 80% to 96% across the three aspects appearance, smell and palate. The SVM baseline performs poorly, achieving around 30% accuracy. The attention-based model achieves reasonable but worse performance than the rationale generator, suggesting the potential of directly modeling rationales as explicit extraction. Figure 5 shows the learning curves of our model for the smell aspect. In the early training epochs, both the independent and (recurrent) dependent selection models fail to produce good rationales, getting low precision as a result. After a few epochs of exploration however, the models start to achieve high accuracy. We observe that the dependent version learns more quickly in general, but both versions obtain close results in the end.
Rationalizing Neural Predictions
Finally we conduct a qualitative case study on the extracted rationales. Figure 3 presents several reviews, with highlighted rationales predicted by the model. Our rationale generator identifies key phrases or adjectives that indicate the sentiment of a particular aspect.
Rationalizing Neural Predictions
Dataset For our second application, we use the real-world AskUbuntu 5 dataset used in recent work (dos Santos et al., 2015;Lei et al., 2016). This set contains a set of 167k unique questions (each consisting a question title and a body) and 16k useridentified similar question pairs. Following previous work, this data is used to train the neural encoder that learns the vector representation of the input question, optimizing the cosine distance (i.e. cosine similarity) between similar questions against random non-similar ones. We use the "one-versusall" hinge loss (i.e. positive versus other negatives) for the encoder, similar to (Lei et al., 2016). During development and testing, the model is used to score 20 candidate questions given each query question, and a total of 400×20 query-candidate question pairs are annotated for evaluation 6 .
Rationalizing Neural Predictions
Task/Evaluation Setup The question descriptions are often long and fraught with irrelevant details. In this set-up, a fraction of the original question text should be sufficient to represent its content, and be used for retrieving similar questions. Therefore, we will evaluate rationales based on the accuracy of the question retrieval task, assuming that better rationales achieve higher performance. To put this performance in context, we also report the accuracy when full body of a question is used, as well as titles alone. The latter constitutes an upper bound on 5 askubuntu.com 6 https://github.com/taolei87/askubuntu  the model performance as in this dataset titles provide short, informative summaries of the question content. We evaluate the rationales using the mean average precision (MAP) of retrieval.
Rationalizing Neural Predictions
Results Table 4 presents the results of our rationale model. We explore a range of hyper-parameter values 7 . We include two runs for each version. The first one achieves the highest MAP on the development set, The second run is selected to compare the models when they use roughly 10% of question text (7 words on average). We also show the results of different runs in Figure 6. The rationales achieve the MAP up to 56.5%, getting close to using the titles. The models also outperform the baseline of using the noisy question bodies, indicating the the models' capacity of extracting short but important fragments. Figure 7 shows the rationales for several questions in the AskUbuntu domain, using the recurrent version with around 10% extraction. Interestingly, the model does not always select words from the question title. The reasons are that the question body can contain the same or even complementary information useful for retrieval. Indeed, some rationale fragments shown in the figure are error messages, i accidentally removed the ubuntu soBware centre , when i was actually trying to remove my ubuntu one applica9ons . although i do n't remember directly uninstalling the centre , i think dele9ng one of those packages might have triggered it . i can not look at history of applica9on changes , as the soBware centre is missing . please advise on how to install , or rather reinstall , ubuntu soBware centre on my computer . how do i install ubuntu soBware centre applica9on ?
Rationalizing Neural Predictions
i know this will be an odd ques9on , but i was wondering if anyone knew how to install the ubuntu installer package in an ubuntu installa9on . to clarify , when you boot up to an ubuntu livecd , it 's got the installer program available so that you can install ubuntu to a drive . naturally , this program is not present in the installed ubuntu . is there , though , a way to download and install it like other packages ? invariably , someone will ask what i 'm trying to do , and the answer … install installer package on an installed system ? what is the easiest way to install all the media codec available for ubuntu ? i am having issues with mul9ple applica9ons promp9ng me to install codecs before they can play my files . how do i install media codecs ? what should i do when i see <unk> report this <unk> ? an unresolvable problem occurred while ini9alizing the package informa9on . please report this bug against the 'update-manager ' package and include the following error message : e : encountered a sec9on with no package : header e : problem with mergelist <unk> e : the package lists or status file could not be parsed or opened . please any one give the solu9on for this whenever i try to convert the rpm file to deb file i always get this problem error : <unk> : not an rpm package ( or package manifest ) error execu9ng `` lang=c rpm -qp --queryformat % { name } <unk> ' '' : at <unk> line 489 thanks conver9ng rpm file to debian fle how do i mount a hibernated par99on with windows 8 in ubuntu ? i ca n't mount my other par99on with windows 8 , i have ubuntu 12.10 amd64 : error moun9ng /dev/sda1 at <unk> : command-line `mount -t `` n[s ' ' -o `` uhelper=udisks2 , nodev , nosuid , uid=1000 , gid=1000 , dmask=0077 , fmask=0177 '' `` /dev/sda1 '' `` <unk> '' ' exited with non-zero exit status 14 : windows is hibernated , refused to mount . failed to mount '/dev/sda1 ' : opera9on not permiAed the n[s par99on is hibernated . please resume and shutdown windows properly , or mount the volume read-only with the 'ro ' mount op9on which are typically not in the titles but very useful to identify similar questions.
Rationalizing Neural Predictions
We proposed a novel modular neural framework to automatically generate concise yet sufficient text fragments to justify predictions made by neural networks. We demonstrated that our encoder-generator framework, trained in an end-to-end manner, gives rise to quality rationales in the absence of any explicit rationale annotations. The approach could be modified or extended in various ways to other applications or types of data.
Rationalizing Neural Predictions
Choices of enc(·) and gen(·). The encoder and generator can be realized in numerous ways without changing the broader algorithm. For instance, we could use a convolutional network (Kim, 2014;Kalchbrenner et al., 2014), deep averaging network (Iyyer et al., 2015;Joulin et al., 2016) or a boosting classifier as the encoder. When rationales can be expected to conform to repeated stereotypical patterns in the text, a simpler encoder consistent with this bias can work better. We emphasize that, in this paper, rationales are flexible explanations that may vary substantially from instance to another. On the generator side, many additional constraints could be imposed to further guide acceptable rationales.
Rationalizing Neural Predictions
Dealing with Search Space. Our training method employs a REINFORCE-style algorithm (Williams, 1992) where the gradient with respect to the parameters is estimated by sampling possible rationales.
Rationalizing Neural Predictions
Additional constraints on the generator output can be helpful in alleviating problems of exploring potentially a large space of possible rationales in terms of their interaction with the encoder. We could also apply variance reduction techniques to increase stability of stochastic training (cf. (Weaver and Tao, 2001;Mnih et al., 2014;).
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Building interpretable systems for natural language understanding is critical in various domains such as healthcare and finance. One direction is generating natural language explanations for prediction [1][2][3][4], which has been shown to be effective as they can not only offer interpretable explanations for back-box prediction but also provide additional information and supervision for prediction [5][6][7]. For example, given a sentence "The only thing more wonderful than the food is the service.", a human annotator may write an explanation like "Positive, because the word 'wonderful' occurs within three words before the term food.", which is much more informative than the label "positive" as it explains how the decision was made. Moreover, the explanation can serve as an implicit logic rule that can be generalized to other instances like "The food is wonderful, I really enjoyed it.".
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
There are recent works [3,4] that study generating natural language explanations for predictions and/or leverage them as additional features for prediction. For example, Camburu et al. [3] learns a language model to generate natural language explanations for the task of natural language inference by training on a corpus with annotated human explanations. Rajani et al. [2] proposes a two-stage framework for commonsense reasoning which first trains a natural language explanation model and then further trains a prediction model with the generated explanations as additional information. These approaches achieve promising performance in terms of both predictive results and explainability. However, a large number of labeled examples with human explanations are required, which is expensive and sometimes impossible to obtain. Therefore, we are looking for an approach that makes effective prediction, offers good explainability, while only requires a limited number of human explanations for training.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In this paper, we propose such an approach. We start from the intuition that the explanationaugmented prediction model is able to provide informative feedback for generating meaningful natural language explanations. Therefore, different from existing work which trains the explanation generation model and the explanation-augmented prediction model in separate stages, we propose to jointly train the two models. Specifically, taking the task of text classification as an example, we propose a principled probabilistic framework for text classification, where natural language Explanations are treated as Latent Variables (ELV). Variational EM [8] is used for the optimization, and only a set of human explanations are required for guiding the explanation generation process.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In the E-step, the explanation generation model is trained to approximate the ground truth explanations (for instances with annotated explanations) or guided by the explanation-augmentation module through posterior inference (for instances without annotated explanations); in the M-step, the explanation-augmented prediction model is trained with high-quality explanations sampled from the explanation generation model. The two modules mutually enhance each other. As human explanations can serve as implicit logic rules, they can be used for labeling unlabeled data. Therefore, we further extend our ELV framework to an Explantion-based Self-Training (ELV-EST) model for leveraging a large number of unlabeled data in the semi-supervised setting.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
To summarize, in this paper we make the following contributions:
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
• We propose a principled probabilistic framework called ELV for text classification, in which natural language explanation is treated as a latent variable. It jointly trains an explanation generator and an explanation-augmented prediction model. Only a few annotated natural language explanations are required to guide the natural language generation process. • We further extend ELV for semi-supervised learning (the ELV-EST model), which leverages natural language explanations as implicit logic rules to label unlabeled data. • We conduct extensive experiments on two tasks: relation extraction and sentiment analysis. Experimental results prove the effectiveness of our proposed approach in terms of both prediction and explainability in both supervised and semi-supervised settings.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
We first discuss the definition of the problem. Given an input sentence x, we aim to predict its label y and generate a natural language (NL) explanation e describing why x is classified as y. Specifically, given a few amount of training example with NL explanation annotation D E = {(x 1 , y 1 , e 1 , ) , ..., (x n , y n , e n , )} and a relatively large set of labeled examples D L = {(x n+1 , y n+1 ) , ..., (x m , y m )}, our goal is to learn: 1) An explanation generation model E θ that parametrize q(e|x, y), which takes a labeled example (x, y) as input and generates a corresponding natural language explanation e, and 2) an explanation-augmented prediction model M φ , which parametrize p(y|x, e) and takes an unlabeled example x and NL explanations (as implicit rules) E to assign a label y to x.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Given labeled data (x, y), we treat the nature language explanation e as a latent variable. For training, we aim to optimize the evidence lower bound (ELBO) of log p(y|x), which can be formulated as:
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
log p(y|x) = log p(e, y|x)de = log q θ (e|x, y) p(e, y|x) q θ (e|x, y) de (1)
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
where q θ (e|x, y) is the variational distribution of posterior distribution p(e|x, y), p(e|x) is the prior distribution of explanation e for instance x, and p φ (y|e, x) is the explanation-augmented prediction model.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Due to the large search space of natural language explanation e, instead of using reprametrization trick used in variational autoencoder [14], here we use the variational-EM algorithm for optimization. The overview of ELV is illuatrated in Figure 1. Note that in our training data, the explanations of a few number of labeled examples (i.e., D E ) are given. Therefore, we first initialize the explanation generation model E θ = q θ (e|x, y) and the prediction model M φ = p φ (y|e, x) by training on D E with maximum likelihood estimation (MLE). Then, we update the aforementioned models by maximizing the log-likelihood log p(y|x) using D L ∪ D E with variational-EM. In the variational E-step, we train the explanation generator to minimize the KL divergence between q θ (e|x, y) and p(e|x, y), which will be detailed in Section 3.3. In the M-step, we fix θ and p(e|x) and update the parameters φ of the prediction model to maximize the log-likelihood log p(y|x).
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
As the core component of our variational EM framework, the explanation generation model is expected to generate "soft" logic rules in the form of natural language explanations. However, training a seq2seq model to generate explanations of high quality from scratch is very challenging. Motivated by the recent finding that pretrained language models encode various types of factual knowledge and commonsense knowledge in their parameters [15][16][17], we employ UniLM [18]-a unified pre-trained language generation model that achieved state-of-the-art performance on many text generation tasks-as the explanation generation model E θ in our framework. Specifically, the explanation generation model takes as input the concatenation of input sentence x and the text description of its corresponding label y to generate an explanation, which explains the label decision in natural language and can be treated as an implicit logic rule that can generalize to other examples.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Note that in the training data, only a small set of labeled examples are provided with explanations. Therefore, in the variational E-step, for labeled data (x, y) without explanations (i.e. D L ), we are trying to use the variational distribution q θ (e|x, y) to approximate the ground truth posterior p(e|x, y), which can be calculated as
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
is parameterized by the prediction model and provides feedback for generating meaningful natural language explanations. We will introduce the detailed parametrization of p(e|x) and p φ (y|x, e) in the M-step.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
For labeled data with explanations (i.e. D E ), we just need to maximize the likelihood of the ground truth explanations. Therefore, the overall objective function of E-step can be summarized as:
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
During M-step, the explanation-augmented prediction model is trained to predict the label of input sentence x with the explanation e generated from the variational distribution q(e|x, y).
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
However, note that the label y is not available during testing, and the explanations for the unlabeled x can only be generated from the prior distribution p(e|x). Therefore, there are some discrepancies between the distributions of the explanations for labeled data in the training stage and those for unlabeled data in the testing stage since generating a natural language explanation without conditioning on a label is harder. To mitigate this issue, in the prediction model, besides sampling an explanation from the variational distribution, we also add a set of explanations from p(e|x), which retrieves a set of explanations from similar sentences.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Specifically, given an input sentence x and the set of labeled and pseudo-labeled data consists of (x ′ , e ′ , y ′ ), we retrieve N explanations E := {e ′ i } N i=1 of which the corresponding sentences x ′ are the most similar to the input sentence x, measured by the cosine similarity between the embedding of x and each x ′ from D E under SentenceBERT [19], a pretrained sentence embedding model. Note that we do not directly use a seq2seq model to parametrize p(e|x) because we find generating explanations without predicted labels often results in irrelevant and even misleading explanations.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
.., e n } denotes all the explanations of x. For each e i ∈ E, we feed the explanation e i and the input sentence x, separated by a [SEP]token, to BERT [20] and use the vector at the [CLS] token to represent the interactions between x and e i as a 768-dimensional feature vector:
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Our final classifier takes the concatenation of these vectors and outputs the final prediction as:
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
At test time, for each unlabeled x, we first use p(e|x) to retrieve a set of explanations and then predict a label with the explanation-augmented prediction model. Afterward, we can further employ the explanation generation model to generate an NL explanation to explain the prediction decision based on both the input sentence and the predicted label.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
To summarize, by alternating between E-step and M-step where q θ (e|x, y) and p φ (y|e, x) are optimized respectively, the explanation generation model E θ and the explanation-augmented prediction model M φ are jointly optimized and mutually enhanced. Next, we describe how our framework can be applied to the semi-supervised setting where both human-annotated explanations and groundtruth labels are limited.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
As natural language explanations can serve as implicit logic rules, which can generalize to new data and help assign pseudo-labels to unlabeled data. Therefore, we extend the ELV to the semi-supervised learning setting and propose an Explanation-based Self-Training (ELV-EST) algorithm. In this setting, we only have limited labeled examples but abundant unlabeled data D U = {x m+1 , . . . , x N }.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
As illustrated in Algorithm 1, we first use ELV to initialize E θ and M φ with the limited labeled corpus D E ∪D L . Afterward, we iteratively use M φ to assign pseudo-labels to unlabeled examples in D U to extend the labeled data D L . We then use ELV to jointly train E θ and M φ with the augmented In return, the newly generated explanations can not only improve the explanation generation model but also serve as implicit rules that help the prediction model assign more accurate pseudo-labels in the next iteration.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
The proposed ELV-EST approach is different from the conventional self-training method in two perspectives. First, in addition to predicting pseudo-labels for unlabeled data, our method also discovers implicit logic rules in the form of natural language explanations, which in return helps the prediction model to better assign noisy labels to the unlabeled data. Second, our approach can produce explainable predictions with E θ . Compared to recent works [7,13] that parse explanations to logic forms, our approach does not require task-specific semantic parsers and matching models, making it task-agnostic and applicable to various natural language understanding tasks with minimal additional efforts.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
We conduct experiments on two tasks: relation extraction (RE) and aspect-based sentiment classification (ASC). For relation extraction we choose two datasets, TACRED [23] and SemEval [21] in our experiments. We use two customer review datasets, Restaurant and Laptop, which are part of SemEval 2014 Task 4 [24] for the aspect-based sentiment classification task. We use the humanannotated explanations collected in [7] for training our explanation-based models.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
We  Table 1.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
We employ BERT-base and UniLM-base as the backbone of our prediction model and explanation generation model, respectively. We select batch size over {32, 64} and learning rate over {1e-5, 2e-5, 3e-5}. The number of retrieved explanations is set to 10 for all tasks. We train the prediction model for 3 epochs and the generation model for 5 epochs in each EM iteration. We use Adam optimizers and early stopping with the best validation F1-score.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In the supervised setting, we compare ELV with the BERT-base baseline that directly fine-tunes the pre-trained BERT-base model on the target datasets. To show the importance of modeling the interactions between the explanation generation model and the explanation-augmented prediction model, we also compare with a variant of our model, which only trains the explanation-augmented prediction module with all the explanations generated from the prior distribution, denoted as ELV (M-step only)). We also compare with some state-of-the-art algorithms on the RE and SA tasks.    In the semi-supervised setting, we compare ELV-EST against several competitive semi-supervised text classification methods including Pseudo-Labeling [29], Self-Training [30], and Data Programming [13] which incorporates NL explanations to perform semi-supervised text classification. Note that all compared model variants incorporate BERT-base as the backbone model.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Results on supervised setting. We first present the results in the supervised setting in Table 2 and 3. ELV significantly outperforms the strong BERT baseline in all four datasets, demonstrating the effectiveness of exploiting NL explanations as additional information for natural language understanding. ELV also consistently outperforms the ELV (M-step only), showing that ELV's variational EM training effectively models the interactions between explanation and prediction. Also, the performance of ELV compares favorably against several competitive recent studies focusing on RE and ASC respectively, further demonstrating the effectiveness of ELV.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Results on semi-supervised setting. The results in the semi-supervised setting are presented in Table 4 and 5. In the semi-supervised scenario, ELV-EST method significantly outperforms various semi-supervised text classification methods, as well as the data programming approach. The latter uses pre-defined rules to parse the NL explanations into logic forms and match unlabeled examples, on all four datasets. The improvement upon the BERT-base + self-training baseline is around 7 points for RE datasets and over 15 points for ASC datasets in terms of F1 score. This demonstrates the effectiveness of ELV-EST in the semi-supervised setting.  Results on explanation generation. We further evaluate the quality of the explanation generation model with human evaluation. We invite 5 graduate students with enough English proficiency to score the explanations generated on the test set with input sentences and the labels predicted by the explanation-augmented prediction module 3 . The annotation scenarios include the explanantions' informativeness (Info.), correctness (Corr.), and consistency (Cons.) with respect to the model prediction. The inner-rater agreement is at 0.51 Kappa score. The details of human evaluation and examples of generated explanations are presented in the Appendix due to space constraints.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
For comparison, we include a fine-tuned UniLM model with annotated NL explanations, as well as two baselines trained from scratch using annotated NL explanations, one with a vanilla transformer model and the other an attention-based LSTM seq2seq model. The results are in Table 6. The explanations generated by our ELV framework are substantially better than those generated by the fine-tuned UniLM model. ELV generates better NL explanations that are relevant to the model's decision-making process, because it models the interactions of the explanation generation model and the prediction model. Performance with corrupted explanations. We first investigate the model performance w.r.t. the quality of the retrieved explanations. We compare with corrupted explanations which randomly replace 80% of the words in the original explanations, results shown in Table 7. The performance with corrupted explanations significantly decreases as expected. The high-quality explanations help the model better generalize while the random ones may confuse the model.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Performance with different numbers of explanations. We then investigate the performances with different amounts of explanations. As illustrated in Figure 2 (left), with as few as 15 annotated explanations, ELV significantly outperforms its counterpart trained without the variational EM framework in the supervised setting. The performance of ELV continues to improve with more explanations but the performance of ELV (M-step only) starts to saturate with 45 explanations, showing the importance of modeling the interactions between the explanation generation model and explanationaugmented prediction model. Similar results are observed in the semi-supervised learning setting in Figure 2 (right).
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Performance with different numbers of labeled/unlabeled data. We also investigate the performance of different models with different proportions of training data. From Figure 3 (left), we can see that ELV consistently outperforms the BERT baseline with different amounts of labeled data. Especially, the improvement is the most significant when only 10% of labeled data is used. This is because human explanations provide additional supervision and can serve as implicit logic rules to help generalization. In the semi-supervised learning setting (Figure 3, right), ELV-EST outperforms traditional self-training methods by a large margin, especially with fewer unlabeled data, further confirming the improved generalization ability from explanations.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In this paper, we propose ELV, a novel framework for training interpretable natural language understanding models with limited human annotated explanations. Our framework treats natural language explanations as latent variables that model the underlying reasoning process to enable interactions between explanation generation and explanation-based classification. Experimental results in both supervised and semi-supervised settings show that ELV can not only make effective predictions but also generate meaningful explanations. In the future, we plan to apply our framework to other natural language understanding tasks. In addition, we plan to test the effectiveness of our framework on other pre-trained language models that are either stronger (e.g., XLNet [31], RoBERTa [32], ALBERT [33], ELECTRA [34], etc.) or more computationally efficient. (e.g., DistilBERT [35], BERT-of-Theseus [36], DeeBERT [37], PABEE [38], etc.)
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
On the other hand, such a system also brings potential risks depending on the quality of the generated natural language explanations. For example, the generated natural language could have certain biases, which have been reported in many natural language understanding systems [39,40]. How to mitigate these risks will be our future work. Another potential risk is that the explanation generation model in our framework generates ad-hoc explanations that are not necessarily informative about how the model makes its predictions, since the model can come up with whatever explanation it thinks would pair with its predicted label. This is a common drawback for current explanation generation models. Our framework partially mitigates this problem since the generated explanations are in return used in the training process of the explanation-augmented classifier through the explanation retrieval process.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In this section we present several human annotated natural language explanations and explanationos generated by the explanation generation model in the ELV framework in different datasets for better understanding of the proposed approach. Label: positive Explanation: The string "favorable" appears no more than 5 words after the term.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
c) The most annoying thing, though, is the fact that the servers seem to be trained to drive revenue. (Term: servers) Label: negative Explanation: The word "annoying" occurs before the term.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
Label: positive Explanation: The terem is followed by "wonderful".
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
b) The falafal was rather over cooked and dried but the chicken was fine. (Term: chicken) Label: positive Explanation: The word "fine" occurs within 3 words after the term.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
c) Service was awful -mostly because staff were overwhelmed on a Saturday night. (Term: staff) Label: negative Explanation: The word "unbearable" occurs within three words after the term. Label: positive Explanation: The string "very nicely" occurs after the term by no more than 6 words.
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
c) The design is awesome, quality is unprecedented. (Term: design) Label: positive Explanation: The word "awesome" is within 2 words after the term. 
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
For human evaluation, we random sample 100 examples in the test set of the Restaurant dataset and use ELV to predict the labels of the selected examples. Then we use different compared models to generate explanations of the prediction results with both the input sentences and the predicted labels. Afterward, we invite 5 graduate students with enough English proficiency to score the explanations. The annotation scenarios include the explanantions' informativeness(Info.), correctness (Corr.), and consistency (Cons.) with respect to the model prediction. Specifically, the informativeness measures to what extent the generated explanation is helpful to understand the model's prediction output. The correctness measures whether the explanation is factually correct (e.g. word "good" before the terms leads to positive label while word "annoying" is negative). The consistency refers to whether the explanation is consistent with the input sentence (i.e. the description in the explanation is true w.r.t the input sentence).
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables
In this section, we report additional experimental results comparing the performance of ELV with 60% of labeled data and the performance with the BERT-base model with 60%, 70%, 80%, 90%, and 100% of labeled data to investigate to what extent human annotated explanations can replace human labeled examples. The result is shown in Table 8. We find that ELV can achieve or even exceed the performance of a BERT-base model trained with much more labeled data. This confirms that ELV can effectively leverage human annotated explanations as additional information.