What input perturbation-based methods have been proposed to evaluate the faithfulness of rationales produced by NLP models?
The majority of existing methods for quantita-
tively evaluating faithfulness is based on input per-
turbation (Nguyen, 2018; DeYoung et al., 2020; Ju
et al., 2022). The main idea is to modify the input
by entirely removing or retaining tokens according
to their FA scores aiming to measure the difference
in predictive likelihood .
Commonly-used perturbation methods include
comprehensiveness, i.e. removing the rationale
from the input), and sufficiency, i.e. retaining only
the rationale (DeYoung et al., 2020). Another com-
mon approach is to remove a number of tokens and
observe the number of times the predicted label
changes, i.e. Decision Flip (Serrano and Smith,
2019). On the other hand, Monotonicity incremen-
tally adds more important tokens while Correla-
tion between Importance and Output Probability
(CORR) continuously removes the most important
tokens (Arya et al., 2021). (In)fidelity perturbs the
input by dropping a number of tokens in a decreas-
ing order of attribution scores until the prediction
changes (Zafar et al., 2021). Additionally, Yin et al.
(2022) proposed sensitivity and stability, which
do not directly remove or keep tokens. Sensitiv-
ity adds noise to the entire rationale set aiming to
find a minimum noise threshold for causing a pre-
diction flip. Stability compares the predictions on
semantically similar inputs
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for generating local explanations. While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable. This paper is a first step towards more robust evaluations of local explanations. We evaluate a variety of local explanation approaches using automatic measures based on word deletion. Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements.
[{"first":"Dong","last":"Nguyen","middle":[]}]
Logic Traps in Evaluating Attribution Scores
Modern deep learning models are notoriously opaque, which has motivated the development of methods for interpreting how deep models predict.This goal is usually approached with attribution method, which assesses the influence of features on model predictions. As an explanation method, the evaluation criteria of attribution methods is how accurately it reflects the actual reasoning process of the model (faithfulness). Meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to demonstrate their arguments.However, some crucial logic traps in these evaluation methods are ignored in most works, causing inaccurate evaluation and unfair comparison.This paper systematically reviews existing methods for evaluating attribution scores and summarizes the logic traps in these methods.We further conduct experiments to demonstrate the existence of each logic trap.Through both theoretical and experimental analysis, we hope to increase attention on the inaccurate evaluation of attribution scores. Moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and starting efforts on reducing the impact of proposed logic traps.
[{"first":"Yiming","last":"Ju","middle":[]},{"first":"Yuanzhe","last":"Zhang","middle":[]},{"first":"Zhao","last":"Yang","middle":[]},{"first":"Zhongtao","last":"Jiang","middle":[]},{"first":"Kang","last":"Liu","middle":[]},{"first":"Jun","last":"Zhao","middle":[]}]
ERASER: A Benchmark to Evaluate Rationalized NLP Models
State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the ‘reasoning’ behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the Evaluating Rationales And Simple English Reasoning (ERASER a benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of “rationales” (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/
[{"first":"Jay","last":"DeYoung","middle":[]},{"first":"Sarthak","last":"Jain","middle":[]},{"first":"Nazneen Fatema","last":"Rajani","middle":[]},{"first":"Eric","last":"Lehman","middle":[]},{"first":"Caiming","last":"Xiong","middle":[]},{"first":"Richard","last":"Socher","middle":[]},{"first":"Byron C.","last":"Wallace","middle":[]}]
Is Attention Interpretable?
Attention mechanisms have recently boosted performance on a range of NLP tasks. Because attention layers explicitly weight input components’ representations, it is also often assumed that attention can be used to identify information that models found important (e.g., specific contextualized word tokens). We test whether that assumption holds by manipulating attention weights in already-trained text classification models and analyzing the resulting differences in their predictions. While we observe some ways in which higher attention weights correlate with greater impact on model predictions, we also find many ways in which this does not hold, i.e., where gradient-based rankings of attention weights better predict their effects than their magnitudes. We conclude that while attention noisily predicts input components’ overall importance to a model, it is by no means a fail-safe indicator.
[{"first":"Sofia","last":"Serrano","middle":[]},{"first":"Noah A.","last":"Smith","middle":[]}]
On the Sensitivity and Stability of Model Interpretations in NLP
Recent years have witnessed the emergence of a variety of post-hoc interpretations that aim to uncover how natural language processing (NLP) models make predictions. Despite the surge of new interpretation methods, it remains an open problem how to define and quantitatively measure the faithfulness of interpretations, i.e., to what extent interpretations reflect the reasoning process by a model. We propose two new criteria, sensitivity and stability, that provide complementary notions of faithfulness to the existed removal-based criteria. Our results show that the conclusion for how faithful interpretations are could vary substantially based on different notions. Motivated by the desiderata of sensitivity and stability, we introduce a new class of interpretation methods that adopt techniques from adversarial robustness. Empirical results show that our proposed methods are effective under the new criteria and overcome limitations of gradient-based methods on removal-based criteria. Besides text classification, we also apply interpretation methods and metrics to dependency parsing. Our results shed light on understanding the diverse set of interpretations.
[{"first":"Fan","last":"Yin","middle":[]},{"first":"Zhouxing","last":"Shi","middle":[]},{"first":"Cho-Jui","last":"Hsieh","middle":[]},{"first":"Kai-Wei","last":"Chang","middle":[]}]
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
As artificial intelligence and machine learning algorithms make further inroads into society, calls are increasing from multiple stakeholders for these algorithms to explain their outputs. At the same time, these stakeholders, whether they be affected citizens, government regulators, domain experts, or system developers, present different requirements for explanations. Toward addressing these needs, we introduce AI Explainability 360 (http://aix360.mybluemix.net/), an open-source software toolkit featuring eight diverse and state-of-the-art explainability methods and two evaluation metrics. Equally important, we provide a taxonomy to help entities requiring explanations to navigate the space of explanation methods, not only those in the toolkit but also in the broader literature on explainability. For data scientists and other users of the toolkit, we have implemented an extensible software architecture that organizes methods according to their place in the AI modeling pipeline. We also discuss enhancements to bring research innovations closer to consumers of explanations, ranging from simplified, more accessible versions of algorithms, to tutorials and an interactive web demo to introduce AI explainability to different audiences and application domains. Together, our toolkit and taxonomy can help identify gaps where more explainability methods are needed and provide a platform to incorporate them as they are developed.
[{"first":"Vijay","last":"Arya","middle":[]},{"first":"Rachel","last":"Bellamy","middle":["K.","E."]},{"first":"Pin-Yu","last":"Chen","middle":[]},{"first":"Amit","last":"Dhurandhar","middle":[]},{"first":"Michael","last":"Hind","middle":[]},{"first":"Samuel","last":"Hoffman","middle":["C."]},{"first":"Stephanie","last":"Houde","middle":[]},{"first":"Q.","last":"Liao","middle":["Vera"]},{"first":"Ronny","last":"Luss","middle":[]},{"first":"Aleksandra","last":"Mojsilovi'c","middle":[]},{"first":"Sami","last":"Mourad","middle":[]},{"first":"Pablo","last":"Pedemonte","middle":[]},{"first":"Ramya","last":"Raghavendra","middle":[]},{"first":"John","last":"Richards","middle":[]},{"first":"Prasanna","last":"Sattigeri","middle":[]},{"first":"Karthikeyan","last":"Shanmugam","middle":[]},{"first":"Moninder","last":"Singh","middle":[]},{"first":"Kush","last":"Varshney","middle":["R."]},{"first":"Dennis","last":"Wei","middle":[]},{"first":"Yunfeng","last":"Zhang","middle":[]}]
On the Lack of Robust Interpretability of Neural Text Classifiers
With the ever-increasing complexity of neural language models, practitioners have turned to methods for understanding the predictions of these models. One of the most well-adopted approaches for model interpretability is feature-based interpretability, i.e., ranking the features in terms of their impact on model predictions. Several prior studies have focused on assessing the fidelity of feature-based interpretability methods, i.e., measuring the impact of dropping the top-ranked features on the model output. However, relatively little work has been conducted on quantifying the robustness of interpretations. In this work, we assess the robustness of interpretations of neural text classifiers, specifically, those based on pretrained Transformer encoders, using two randomization tests. The first compares the interpretations of two models that are identical except for their initializations. The second measures whether the interpretations differ between a model with trained parameters and a model with random parameters. Both tests show surprising deviations from expected behavior, raising questions about the extent of insights that practitioners may draw from interpretations.
[{"first":"Muhammad","last":"Zafar","middle":["Bilal"]},{"first":"Michele","last":"Donini","middle":[]},{"first":"Dylan","last":"Slack","middle":[]},{"first":"C'edric","last":"Archambeau","middle":[]},{"first":"Sanjiv","last":"Das","middle":[]},{"first":"Krishnaram","last":"Kenthapadi","middle":[]}]
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
While the impact of machine learning is increasing rapidly in society, machine learning systems have also become increasingly complex and opaque. Classification models are usually evaluated based on prediction performance alone (e.g., by measuring the accuracy, recall, and precision) and the interpretability of these models has generally been undervalued. However, the importance of interpretable models is increasingly being recognized (Doshi-Velez and Kim, 2017;Freitas, 2014).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
First, higher interpretability could lead to more effective models by revealing incompleteness in the problem formalization (Doshi-Velez and Kim, 2017), by revealing confounding factors that could lead to biased models, and by supporting error analyses or feature discovery (Aubakirova and Bansal, 2016). Second, with the increasing adoption of machine learning approaches for humanities and social science research, there is also an increasing need for systems that support exploratory analyses and theory development.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Various approaches have been explored to increase the interpretability of machine learning models (Lipton, 2016). This paper focuses on local explanation, which aims to explain the prediction for an individual instance (e.g., Ribeiro et al. (2016)). A study by Herlocker et al. (2000) found that providing local explanations could help improve the acceptance of movie recommendation systems. Local explanations can come in different forms. For example, Koh and Liang (2017) identify the most influential training documents for a particular prediction. The most common type of local explanation involves identifying the important parts of the input for a prediction, such as the most predictive words in a document for a text classification model.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
In this paper we focus on local explanations for text classification. Below is a fragment of a movie review. The words identified by a local explanation method to explain a neural network prediction are in bold. The review is labeled with a negative sentiment, but the classifier incorrectly predicted a positive sentiment. The highlighted words help us understand why. steve martin is one of the funniest men alive. if you can take that as a true statement, then your disappointment at this film will equal mine. martin can be hilarious, creating some of the best laugh-out-loud experiences that have ever taken place in movie theaters. you won't find any of them here. [...] Words such as funniest and hilarious were important for the prediction. Besides providing evidence for a predicted label, some local explanations can also provide evidence against a predicted label. For example, in the above example, the word disappointment was one of the highest ranked words against the predicted label.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Ineffective approaches could generate misleading explanations (Lipton, 2016), but evaluating local explanations is challenging. A variety of approaches has been used, including only visual inspection (Ding et al., 2017;Li et al., 2016a), intrinsic evaluation approaches such as measuring the impact of deleting the identified words on the classifier output (Arras et al., 2016), and user studies (Kulesza et al., 2015).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Contributions To further progress in this area, it is imperative to have a better understanding of how to evaluate local explanations. This paper makes the following contributions:
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• Comparison of local explanation methods for text classification. We present an in-depth comparison between three local explanation approaches (and a random baseline) using two different automatic evaluation measures on two text classification tasks (Section 4).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• Automatic versus human evaluation. Automatic evaluations, such as those based on word deletions, are frequently used since they enable rapid iterations and are easy to reproduce. However, it is unclear to what extent they correspond with human-based evaluations. We show that the automatic measures correlate moderately with human judgements in a task setting and that other factors also impact human judgement. (Section 5).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
This section describes the datasets, the classification models and the local explanation approaches used in our experiments.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
We experiment with two datasets (Table 1):
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• Twenty newsgroups (20news). The Twenty Newsgroups dataset has been used in several studies on ML interpretability (Arras et al., 2016;Kapoor et al., 2010;Ribeiro et al., 2016). Similar to Ribeiro et al. (2016), we only distinguish between Christianity and Atheism. We use the 20news-bydate version, and randomly reserve 20% of the training data for development.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• Movie reviews. Movie reviews with polarity labels (positive versus negative sentiment) from Pang and Lee (2004). We use the version from Zaidan et al. (2007). The dataset is randomly split into a train (60%), development (20%) and test (20%) set. 
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
We experiment with two different models. Logistic Regression (LR) is implemented using Scikitlearn (Pedregosa et al., 2011) with Ridge regularisation, unigrams and a TF-IDF representation, resulting in a 0.797 accuracy on the movie dataset and a 0.921 accuracy on the 20news dataset. We experiment with a LR model, because the contributions of individual features in a LR model are known. We thus have a ground truth for feature importance to compare against for this model. We also use a feedforward neural network (MLP) implemented using Keras (Chollet et al., 2015), with 512 hidden units, ReLU activation, dropout (0.5, not optimized) and Adam optimization, resulting in a 0.832 accuracy on the movie dataset and a 0.939 accuracy on the 20news dataset.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
In this paper, we focus on local explanation approaches that identify the most influential parts of the input for a particular prediction. In this paper we limit our focus to individual words for explaining the output of text classification models. Other representations, e.g., explanations using phrases or higher-level concepts are left for future work. We experiment with explanations for the predicted class, since in real-life settings usually no ground truth labels are available. We experiment with the following local explanation approaches:
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• Random. A random selection of words in the document.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• LIME (Ribeiro et al., 2016) is a modelagnostic approach and involves training an interpretable model (in this paper, a linear model with Ridge regularisation) on samples created around the specific data point by perturbing the data. We experiment with 500-5000 samples and use the implementation provided by the authors. 1
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• Word omission. This approach aims to estimate the contribution of individual words by deleting them and measuring the effect, e.g., by the difference in probability (Robnik-Sikonja and Kononenko, 2008). Within NLP, variations have been proposed by Kádár et al. (2016), Li et al. (2016b) and Martens and Provost (2014). It is also similar to occlusion in the context of image classification, which involves occluding regions of the input image (Zeiler and Fergus, 2014). For LR, this approach corresponds to ranking words according to the regression weights (and considering the frequency in the text) and is therefore optimal. For MLP, we use the difference in probability for the predicted class (ŷ) when removing word w from input x: p(ŷ|x) − p(ŷ|x \w ). This approach supports explanations based on interpretable features (e.g., words) even when the underlying representation may be less interpretable. However note that in general, this omission approach might not be optimal, since it estimates the contribution of words independently. This approach is also computationally expensive, especially when many features are used.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
• First derivative saliency. This approach computes the gradient of the output with respect to the input (e.g., used in Aubakirova and Bansal (2016), Li et al. (2016a) and Simonyan et al. (2013)). The obtained estimates are often referred to as saliency values. Several variations exist, e.g., Li et al. (2016a) take the absolute value. In this paper, the raw value is taken to identify the words important for and against a certain prediction.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
In this section we explore automatic evaluation of local explanations. Local explanations should exhibit high local fidelity, i.e. they should match the underlying model in the neighborhood of the instance (Ribeiro et al., 2016). An explanation with low local fidelity could be misleading. Because we generate explanations for the predicted class (rather than the ground truth), explanations with high local fidelity do not necessarily need to match human intuition, for example when the classifier is weak . Ideally, the evaluation metrics are model agnostic and do not require information that may not always be available such as probability outputs. This paper focuses on local fidelity, but other aspects might also be desired, such as sparsity Ribeiro et al., 2016;Martens and Provost, 2014).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
We measure local fidelity by deleting words in the order of their estimated importance for the prediction. Arras et al. (2016) generated explanations with the correct class as target. By deleting the identified words, accuracy increased for incorrect predictions and decreased for correct predictions. However, their approach assumes knowledge of the ground-truth labels. We take an alternative, but similar, approach. Words are also deleted according to their estimated importance, e.g. w 1 ...w n with w 1 the word with the highest importance score, but for the predicted class instead. For each document, we measure the number of words that need to be deleted before the prediction switches to another class (the switching point), normalized by the number of words in the document. For example, a value of 0.10 indicates that 10% of the words needed to be deleted before the prediction changed. An advantage of this approach is that ground-truth labels are not needed and that it can be applied to blackbox classifiers, we only need to know the predicted class. Furthermore, the approach acts on the raw input. It requires no knowledge of the underlying feature representation (e.g., the actual features might be on the character level). We also experiment with the measure proposed by , referred to as the area over the perturbation curve (AOPC):
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
is the probability for the predicted class when words 1..k are removed and · p(x) denotes the average over the documents. This approach is also based on deleting words, but it is more fine-grained since it uses probability values rather than predicted labels. It also enables evaluating negative evidence. A drawback is that AOPC requires access to probability estimates of a classifier. In this paper, K is set to 10.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
For LR, the exact contribution of individual features to a prediction is known and the words in the document that contributed most to the prediction can be computed directly. For this classifier, the optimal approach corresponds to the omission approach. Table 3 reports the results by measuring the effect of word deletions and reporting the average switching point. Lower values indicate that the method was better capable of identifying the words that contributed most towards the predicted class, because on average fewer words needed to be deleted to change a prediction. Table 2 shows the AOPC values with a cut-off at 10. We measure AOPC in two settings: removing positive evidence (higher values indicate a more effective explanation) and negative evidence (lower values indicate a more effective explanation).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Comparison local explanation methods As expected, LIME improves consistently when more samples are used. Furthermore, when comparing the scores of the omission approach for the LR model (which corresponds to the ground-truth) we observe that LIME with 5000 samples comes close to the optimal score. We use the two-tailed paired permutation test to test for significance between all methods with both evaluation measures. In al-    most all cases, the differences are highly significant (p < 0.001), except the difference in average switching point between the omission and salience approach on the movies dataset with the MLP classifier (n.s.) and the difference in average switching point between the omission and LIME-5000 approach on 20news with the MLP classifier (n.s.). The difference in AOPC scores for evaluating negative evidence was not significant in many cases.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Metric sensitivity First, the results suggest that the values obtained depend strongly on the type of task and classifier. The explanation approaches score better on the sentiment detection task in both Tables 2 and 3. For example, fewer words need to be removed on average to change a prediction in the movie dataset (Table 3). A possible explanation is that for sentiment detection, a few words can provide strong cues for the sentiment (e.g., terrific), while for (fine-grained) topic detection (e.g., distinguishing between Christianity and atheism) the evidence tends to be distributed among more words. Better values are also obtained for the LR classifier (a linear model) than for MLP.  Second, as shown in Table 2, AOPC enables assessing negative evidence (i.e. the words that provide evidence for the opposite class). The obtained absolute values are much smaller compared to the values obtained for the words identified as positive evidence. This is expected, since the positive evidence in a document for the predicted class should be larger than the negative evidence.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Third, we analyze the relation between the word deletion evaluation measures and the prediction confidence of the classifiers, based on the probability of the output class. Table 4 reports the Spearman correlations for the MLP classifier on the movie dataset (similar trends were observed with the LR classifier). There is a strong correlation between the prediction confidence and the word deletion evaluation measures. The higher the prediction confidence of a classifier, the more words need to be deleted before a prediction changes (e.g., see the switching points). However, the strength of the correlations is lower for the more robust explanation methods (LIME-5000, omission and saliency).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
In the previous section we evaluated the local explanation approaches using automatic measures. However, the explanations are meant to be presented to humans. We therefore turn to evaluating the explanations using crowdsourcing. We analyze the usefulness of the generated explanations in a task setting and analyze to what extent the automatic measures correspond to the human-based evaluations. The crowdsourcing experiments are run on CrowdFlower. Only crowdworkers from Australia, Canada, Ireland, United Kingdom and the United States and with quality levels two or three were accepted.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
One way to evaluate an explanation is by asking humans to guess the output of a model based on the explanation and the input. Doshi-Velez and Kim (2017) refer to this as forward simulation/prediction. As mentioned by Doshi-Velez and Kim (2017), this is a simplified task. Evaluations using more specific application-oriented tasks or tailored towards specific user groups should be explored in future work. We have chosen the forward prediction task as a first step since it is a general setup that could be used to evaluate explanations for a variety of tasks and models.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
In this study, crowdworkers are shown the texts (e.g., a movie review), in which the top words identified by the local explanation approaches are highlighted. Crowdworkers are then asked to guess the output of the system (e.g., a positive or negative sentiment). The crowdworkers are also asked to state their confidence on a five-point Likert scale ('I am confident in my answer': strongly disagree . . . strongly agree).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Note that the workers need to guess the output of the model regardless of the true label (i.e. the model may be wrong). The crowdworkers are therefore presented with documents with different prediction outcomes (true positive, true negative, false negative, and false positive). We sample up to 50 documents for each prediction outcome. A screenshot is shown in Figure 1. A quiz and test questions are used to ensure the quality of the crowdworkers. Instructions as well as the test questions included cases where the system made an incorrect prediction, so that workers understood that the task was different than standard labeling tasks. See Appendix A for more details.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
We experiment with the following parameters: methods (random baseline, LIME with 500 and 5000 samples, word omission, saliency) and the number of words (10, 20). We experiment with both datasets. Due to space constraints, we only experiment with the MLP classifier. We collected the data in August and September 2017. Each HIT (Human Intelligence Task) was carried out by five crowdworkers. We paid $0.03 per judgement. On the 20news dataset, we collected 7,200 judgements from 406 workers (mean nr of. judgements per worker: 17.73, std.: 7.21) and on the movie dataset we collected 8,100 judgements from 445 workers (mean nr of. judgements per worker 18.20, std: 7.24).  Table 6 reports the confidence scores by method. On the movie dataset, the trends match the intrinsic evaluations closely. The random method leads to the lowest confidence score, followed by LIME-500 and LIME-5000, and explanations from the omission and saliency approach both lead to the highest confidence scores. On the 20news dataset, the trends are less clear. We observe a small, significant negative correlation between confidence values and time spent (Spearman correlation: ρ=-0.08, p <0.0001 on the movie dataset, ρ=-0.06, p <0.0001 on 20news).
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Accuracy Table 6 also reports the fraction of correct guesses per method. Random explanations lead to the lowest accuracies, followed by LIME with 500 samples. The differences between LIME-5000, omission and saliency are small and not consistent across datasets. The crowd had a higher accuracy on the movie data, except when the explanations were randomly generated.    Table 5 separates the results by the different prediction outcomes. The results suggest that false positive and false negative are the most revealing. In these cases, crowdworkers are not able to rely on their intuition and a strong explanation should convince them that the system makes a mistake. Otherwise, crowd workers might choose the label matching the document (and not necessarily the classifier output). This is especially salient in the 20news dataset, where the random approach performs better than expected on the true positives and true negatives. For example, compare the random approach with the omission approach on true positives with ten word explanations.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Our experiments also show that local explanations in the form of the most predictive words are sometimes not enough to simulate the output of a system. For example, the best accuracy on true positive instances in the 20news data is only 0.752. The movie dataset contains difficult instances as well. For example, the omission method identifies the following words in a movie review to explain a false positive prediction: 'believes', 'become ', 'hair', 'unhappy', 'quentin', 'directed', 'runs', 'filled', 'fiction', 'clint'. Due to the composition of the training data, the system has associated words like 'quentin' and 'clint' with a positive sentiment. This may have confused the crowdworkers as most of them guessed incorrectly. Expanding the explanation with for example influential documents (Koh and Liang, 2017) or a visualization of the class distributions of the most influential words could make the explanations more informative.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Correlation with automatic evaluation For each explanation, we compute the fraction of workers who correctly predicted the classifier output (the 'crowd accuracy') and correlate these with the automatic measures. We expect a negative correlation with the switching points and a positive correlation with the AOPC. The correlations are moderate (Table 8). The correlations with AOPC on the movie data are the biggest on the false positives and false negatives, when workers are not able to rely on their intuition. The correlations     measured on the true positives in 20news are opposite of what we expect. The 20news data is noisy and the classifier picks up on spurious features, possibly confusing the workers. An example in the 20news data is an email with the following words highlighted: 'thank', 'mail', 'discussions', 'seminary', 'before', 'thanks', 'question', 'fill', 'affected', 'during', 'proofs'. The classifier was confident and the computed switchpoint was low. The e-mail comes from the atheism newsgroup, which becomes clear from reading the text. The highlighted words are all more likely to occur in the christianity newsgroup, but on their own they are not intuitive to lay people. Consequently, workers guessed incorrectly that the predicted label was atheism. Explanations that also show the negative evidence (in this case, words such as 'atheism' and 'atheists') and/or the word distributions across classes would likely have led to better crowd accuracy.  As shown in section 4, the automatic measures correlate strongly with the prediction confidence of the classifier. More words need to be removed before a prediction changes (i.e. a higher switching point) when the classifier is more confident. However, we also find that higher classifier confidence leads to higher crowd accuracies (e.g., ρ = 0.236, p < 0.001 on the 20news dataset). We therefore fit an Ordinary Least Squares (OLS) model to control for these different factors (Table  9), with crowd accuracy as the dependent variable. A higher switching point significantly leads to a lower accuracy. However, classifier confidence and prediction outcome also significantly impact the accuracy. Similar trends are observed for the AOPC measure (Table 10). We also find that the automatic evaluation measures significantly impact crowd accuracy on the 20news dataset, but the patterns are less strong.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Noise In our final experiment we analyze the effect of noise. We focus on explanations based on saliency scores on the movie dataset. We experiment with introducing noise to the top ten words (Table 7) and we collect additional judgements. A noise level of 0.2 indicates that two out of the top ten words are randomly replaced by other words. The results show that with increasing the noise, as expected, both the performance and average AOPC score decrease.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
There has been an increasing interest in improving the interpretability of machine learning systems, but evaluating the quality of explanations has been challenging. This paper focused on evaluating local explanations for text classification. Local explanations were generated by identifying important words in a document for a prediction. We compared automatic evaluation approaches, based on measuring the effect of word deletions, with human-based evaluations. Explanations generated using word omissions and first derivatives both performed well. LIME (Ribeiro et al., 2016) performed close to these methods when using enough samples. Our analyses furthermore showed that the evaluation numbers depend on the task/dataset and the confidence of the classifiers.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Next, crowd workers were asked to predict the output of the classifiers based on the generated explanations. We found moderate, but significant, correlations between the automatic measures and crowd accuracy. In addition, the human judgements were impacted by the confidence of the classifier and the type of prediction outcome (e.g., a false negative versus a true positive). Our results also suggest that only highlighting words is sometimes not enough. An explanation can highlight the most important parts of an input and score well on automatic measures, but if the explanation is not intuitive (for example due to biases in the data), humans are still not able to predict the output.
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
For the classification tasks in this paper (topic classification and sentiment detection) individual words are often predictive. As a result, local explanation approaches that select words independently worked well. However, we expect that for tasks where individual words are not predictive, the current evaluation methods and local explanation approaches may not be sufficient. Furthermore, in future work more fine-grained visualizations (e.g., Handler et al. (2016)) could be explored.
Logic Traps in Evaluating Attribution Scores
The opaqueness of deep models has grown in tandem with their power (Doshi-Velez and Kim, 2017), which has motivated efforts to interpret how these black-box models work (Sundararajan et al., 2017;Belinkov and Glass, 2019). Post-hoc explanation aims to explain a trained model and reveal how the model arrives at a decision (Jacovi and Goldberg, 2020;Molnar, 2020). This goal is usually approached with attribution method, which assesses the influence of features on model predictions as shown in Figure 1. Recent years have witnessed an increasing number of attribution meth-ods being developed. For example, Erasure-based method calculate attribution scores by measuring the change of output after removing target features (Li et al., 2016;Feng et al., 2018;; Gradient-based method uses gradients to study the influence of features on model predictions (Sundararajan et al., 2017;Hao et al., 2020); Meanwhile, these methods also received much scrutiny, arguing that the generated attribution scores are fragile or unreliable (Alvarez-Melis and Jaakkola, 2018; Pruthi et al., 2020;Wang et al., 2020;Slack et al., 2020). As an explanation method, the evaluation criteria of attribution methods should be how accurately it reflects the true reasoning process of the model (faithfulness), not how convincing it is to humans (plausibility) Jacovi and Goldberg (2020). Meanwhile, since the reasoning process of deep models is inaccessible, researchers design various evaluation methods to support their arguments, some of which appear valid and are widely used in the research field. For example, meaningful perturbation is used for making comparison in many works (Samek et al., 2016;Chen et al., 2019;DeYoung et al., 2020;Kim et al., 2020). The philosophy of meaningful perturbation is simple, i.e., modifications to the input instances, which are in accordance with the generated attribution scores, can bring about significant differences to model predictions if the attribution scores are faithful to the target system. However, some crucial logic traps existing in these evaluation methods are ignored in most works, causing inaccurate evaluation and unfair comparison. For example, we found that we can manipulate the evaluation results when using meaningful perturbation to make comparisons: by choosing the modification strategy, we can assign any of the three candidate attribution methods as the best method. The neglect of these traps has damaged the community in many aspects: First, the existence of logic traps will lead to an inaccurate evaluation and unfair comparison, making the conclusions unreliable; Second, the wide use of evaluation metrics with logic traps brings pressure to newly proposed works, requiring them to compare with other works using the same metrics; Last, the over-belief in existing evaluation metrics encourages efforts to propose more accurate attribution methods, notwithstanding the evaluation system is unreliable.
Logic Traps in Evaluating Attribution Scores
In this paper, we systematically review existing methods for evaluating attribution scores and categorize them into classes. We summarize the logic traps in these methods and further conduct experiments to demonstrate the existence of each logical trap. Though strictly accurate evaluation metrics for attribution methods might be a "unicorn" which will likely never be found, we should not just ignore logic traps in existing evaluation methods and draw conclusions recklessly. Through both theoretical and experimental analysis, we hope to increase attention on the inaccurate evaluation of attribution scores. Moreover, with this paper, we suggest stopping focusing on improving performance under unreliable evaluation systems and starting efforts on reducing the impact of proposed logic traps.  is better than others because only it can identify 'favorite' as positive and 'used to be' as negative, which is consistent with human understandings.
Logic Traps in Evaluating Attribution Scores
Furthermore, resorting to human-annotated explanations, works can also evaluate attribution methods quantitatively in evaluation 1. For example, the SST-2 (Socher et al., 2013) corpus provides not only sentence-level labels, but also five-class word-level sentiment tags ranging from very negative to very positive. Thus, many works (Lei et al., 2016;Li et al., 2016;Tsang et al., 2020;Kim et al., 2020) perform quantitative evaluation of attribution scores by comparing them with the word-level tags in SST-2.
Logic Traps in Evaluating Attribution Scores
Logic Trap 1: The decision-making process of neural networks is not equal to the decisionmaking process of humans.
Logic Traps in Evaluating Attribution Scores
First, we cannot completely deny the rationality of evaluation 1. Since many attribution methods work without any human-annotated information, such as erasure-based and gradient-based methods, the similarity between human-annotated explanations and generated attribution scores can be seen as drawing from the reasoning process of target models. However, since the deep model often rely on unreasonable correlations, even when producing correct predictions, attribution scores preposterous to humans may reflect the reasoning process of the deep model faithfully. Thus we cannot deny the validity of an attribution score through its inconsistency to human-annotated explanations and cannot use human-annotated explanations to conduct a quantitative evaluation.
Logic Traps in Evaluating Attribution Scores
In experiment 1, we give an example to demonstrate that the model might rely on correlations inconsistent with human understandings to get the prediction: though trained with questions, a question-answering model could maintain the same prediction for a large ratio of samples when the question information is missing, which is obviously different from humans.
Logic Traps in Evaluating Attribution Scores
We experiment on RACE (Lai et al., 2017), a large-scale question-answering dataset. As shown in Table 2, RACE requires the model to choose the right answer from candidate options according to the given question and document. Document: "...Many people optimistically thought industry awards for better equipment would stimulate the production of quieter appliances. It was even suggested that noise from building sites could be alleviated ..." Question: What was the author's attitude towards the industry awards for quieter?  We first train a model with BERT base (Devlin et al., 2019) as encoder 1 with questions, and achieve 65.7% accuracy on the development set. Then, we replace the development set questions with empty strings and feed them into the trained model. Surprisingly, the trained MRC model maintained the original prediction on 64.0% of the test set samples (68.4% on correctly answered samples and 55.4% on wrongly answered samples). Moreover, we analyze the model confidence change in these unchanged samples, where the probability on the predicted label is used as the confidence score. As shown in Figure 2, most of the samples have confidence decrease smaller than 0.1, demonstrating question information are not essential for the model to get predictions in these samples. Since question information is usually crucial for humans to answer the question, attribution scores faithfully reflect the reasoning process of this model may be inconsistent with human annotations. Thus, it is improper to use human-annotation explanations as the ground truth to evaluate attribution methods.
Logic Traps in Evaluating Attribution Scores
Most existing methods for quantitatively evaluating attribution scores can be summarized as evaluations based on meaningful perturbation. The philosophy of meaningful perturbation is simple, i.e., modifications to the input instances, which are in accordance with the generated attribution scores, can bring about significant differences to the target model's predictions if the attribution scores are faithful to the target model.
Logic Traps in Evaluating Attribution Scores
For example, Samek et al. (2016); Nguyen (2018); Chen and Ji (2020) use the area over the perturbation curve (AOPC) (Samek et al., 2016) as evaluation metrics. Specifically, given the attribution scores of a set of features, AOPC(k) modifies the top k% features and calculates the average change in the prediction probability as follows,
Logic Traps in Evaluating Attribution Scores
whereŷ is the predicted label, N is the number of examples, p(ŷ|) is the probability on the predicted class, andx  Besides these works, a lot of works (Shrikumar et al., 2017;Chen et al., 2019;Nguyen, 2018;DeYoung et al., 2020;Hao et al., 2020;Jiang et al., 2021) use similar metrics to perform evaluation and comparisons. The main difference between evaluation metrics in these works is the difference in the modification strategy. For example, to evaluate word-level attribution scores for SST-2,  uses deleting tokens as modification while Kim et al. (2020) uses replacing tokens with tokens sampled from the distribution inferred by BERT.
Logic Traps in Evaluating Attribution Scores
Logic Trap 2: Using an attribution method as the ground truth to evaluate the target attribution method.
Logic Traps in Evaluating Attribution Scores
Evaluation methods based on meaningful perturbation can be seen as an attribution method too. For example, AOPC(k), which assesses the importance of k% features, can be seen as an attribution method calculating an attribution score for k% features. Specifically, when using deleting tokens as modification and narrowing the k% to one token, AOPC(k) degenerates into the basic attribution method: leave-one-out (Li et al., 2016). Thus, evaluation 2 uses an attribution method as the ground truth to evaluate the target attribution method, which measures the similarity between two attribution methods instead of faithfulness.
Logic Traps in Evaluating Attribution Scores
Since meaningful perturbation assesses the importance of features by calculating output change after modifications, its results are mainly depend on how to conduct the modifications, which means different modification strategies might lead to different evaluation results. Evaluation 2 is widely used to compare attribution methods in the research field. Accordingly, the neglect of logic trap 2 has led to a high risk of unfair comparisons and unreliable conclusions.
Logic Traps in Evaluating Attribution Scores
In experiment 2, we give an example of unfair comparisons in evaluation 2: the more similar the target attribution method to the modification strategy, the better the evaluation results. Specifically, by modifying the modification strategies in APOC and AUC, we can assign any of the three candidate attribution methods as the best method. We conduct experiments on on widely used SST-2 task of the GLUE benchmark ), and use BERT base as encoder to build the target model 1 (achieve 86.4% accuracy).
Logic Traps in Evaluating Attribution Scores
Attribution Methods We experiment with three attribution methods: leave-one-out (LOO) (Li et al., 2016), HEDGE  and Marg (Kim et al., 2020). The schemes of these attribution methods are shown in Figure 4, LOO assign attribution scores to the target word 'good' by deleting it from the sentence and observing change in the model predictions; Marg marginalizes the target word 'good' out considering the likelihoods of all candidate words, which uses BERT to measure the likelihoods of candidate words to replace the target word; HEDGE builds hierarchical explanations by recursively detecting the weakest interactions and then dividing large text spans into smaller ones. HEDGE assign attribution scores to spans by using '[PAD]' token to replace other words in a sentence and measuring how far the prediction is to the prediction boundary.
Logic Traps in Evaluating Attribution Scores
We first evaluate three attribution methods with metrics drawn from Marg and HEDGE papers. Marg uses AUC as evaluation metrics and modifies words by gradually replacing them with a token sampled from the distribution inferred by BERT, denoted as AUC rep ; HEDGE uses AOPC as evaluation metrics and modifies words by deleting them directly, denoted as AOPC del . Both papers modify 20% of words in the sentence. The results are shown in Table 3. As shown in Table 3  modification strategy of AOPC del is consistent with LOO, and that of AUC rep is most similar to Marg, the evaluation results are consistent with the inference in logic trap 2: the more similar the target evaluated method to the modification strategy, the better the evaluation results.
Logic Traps in Evaluating Attribution Scores
We further conduct ablation experiments by changing the modification strategies in AOPC del and AUC rep . Concretely, we switched perturbing strategy in AOPC del and AUC rep and get new evaluation metrics: AOPC rep and AUC del . As shown in Table  3, different from the initial results, Marg performs best in APOC metric while LOO performs best in AUC metric. The opposite results demonstrate that evaluation results mainly depend on the modification strategies, and we can manipulate evaluation results by changing them. Moreover, we note that HEDGE performs worst in all four evaluation metrics. Thus, we further customize the modification strategy to HEDGE's advantage: padding unimportant features according to the attribution scores, denoted as AOPC pad and AUC pad . Not surprisingly, results in Table 3 show that HEDGE perform best in customized metrics.
Logic Traps in Evaluating Attribution Scores
Because of the existence of logic trap 2, we can manipulate the evaluation results in evaluation 2 by changing the modification strategies, assigning any of the three candidate attribution methods as the best method. In fact, because we cannot simply assign a modification strategy as faithful, we should not use evaluation 2 to quantitatively evaluate attribution scores and make comparisons. Since the wide use of evaluation 2, the neglect of logic trap 2 has negatively impacted the research field for a long time. First, it brings a risk of unfair comparisons: works can customize evaluation metrics to their advantage and thus achieve the best performance. Second, the wide use of evaluation 2 also brings pressure to new proposed works, forcing them to make comparisons to others in such evaluation.
Logic Traps in Evaluating Attribution Scores
In this evaluation, works evaluate attribution methods by examining the consistency of attribution scores for similar inputs. The philosophy of Evaluation 3 is that semantically similar inputs which share the same model predictions should have similar attribution scores if the attribution method is reliable. Evaluation 3 is often used to disprove the effectiveness of attribution methods by searching for counterexamples.
Logic Traps in Evaluating Attribution Scores
For example, ExplainFooler (Sinha et al., 2021) attacks Integrated Gradients and (Sundararajan et al., 2017) and LIME (Sundararajan et al., 2017), which are two popular attribution methods in NLP, by searching adversarial sentences with different attribution scores. As shown in Figure 5, these adversarial sentences are semantically similar to the original sentence and share the same model predictions. However, the attribution scores of these sentences are very different from that of the original sentence. Sinha et al. (2021) observes the rank order correlation drops by over 20% when less than 10% of words are changed on average and thus draws the conclusion that Integrated Gradients and LIME are fragile. A lot of works (Alvarez-Melis and Jaakkola, 2018; Kindermans et al., 2019;Ghorbani et al., 2019;Ding and Koehn, 2021;Sinha et al., 2021) use evaluation 3 to examine the validity of existing attribution methods. For example, Ghorbani et al. (2019) argues that interpretations of neural networks are fragile by showing that systematic perturbations can lead to different interpretations without changing the label; Alvarez-Melis and Jaakkola (2018) argues that a crucial property that attribution methods should satisfy is robustness to local perturbations of the input.
Logic Traps in Evaluating Attribution Scores
Logic Trap 3: The change in attribution scores maybe because the model reasoning process is really changed rather than the attribution method is unreliable.
Logic Traps in Evaluating Attribution Scores
When solving similar samples like those shown in Figure 5, humans tend to use similar reasoning methods. However, deep models are not as robust enough as humans and often rely on unreasonable correlations. Semantically similar texts often cause different reasoning processes in deep models. For example, it is well known that deep models are vulnerable to adversarial samples (Goodfellow et al., 2015;Papernot et al., 2016). By deliberately adding some subtle interference that people cannot detect to the input sample, the target model will give a different prediction with high confidence. The success in adversarial attacks on deep models demonstrates similar inputs for humans can share very different reasoning processes in deep models.
Logic Traps in Evaluating Attribution Scores
The main difference between attributionattacking methods and model-attacking is that attribution-attacking methods require the model to give the same prediction for adversarial samples. However, giving the same prediction is very weak to constraint model reasoning because deep models have compressed the complicated calculation process into limited classes in the prediction. For example, there is always half probability of giving the same prediction for a binary classification task even with totally random reasoning. Thus, it is no surprise that attribution-attacking methods can find adversarial samples which share the same prediction label to the original sample yet have different attribution scores.
Logic Traps in Evaluating Attribution Scores
The logic trap in evaluation 3 is that the change in attribution scores may be because the model reasoning process is really changed rather than the attribution method is unreliable. As shown in Fig-Figure 6: We use lines connecting inputs and outputs to represent the model reasoning process. (a) is a successful attack on the target model while (b) might be regarded as a successful attack on attribution methods, falling into the logic trap 3. ure 6. (b), an attribution method should generate different attribution scores for the original and adversarial samples if it faithfully reflects the model reasoning. However, it will be regarded as fragile or unreliable in evaluation 3. Unfortunately, existing works ignore this logic trap and propose various methods to attack attribution methods. Since the high susceptibility of deep models to adversarial samples, not surprisingly, all of these works get the same conclusion: existing attribution methods are fragile or unreliable.
Logic Traps in Evaluating Attribution Scores
In experiment 3, we demonstrate that deep models can assign the same label to semantically similar samples yet use different reasoning. We experiment on widely used SST-2 and MNLI tasks of the GLUE benchmark ). MNLI requires the model to predict whether the premise entails the hypothesis, contradicts it, or is neutral.
Logic Traps in Evaluating Attribution Scores
Model Since the attribution methods are defaulted as unreliable in evaluation 3, we cannot use existing attribution methods to judge whether the model reasoning is different. To solve the problem, we use a two-stage model framework, where the model first extracts a subset of inputs and gives prediction based only on the subset information. This way, we can observe whether the model reasoning is changed from the chosen subset, i.e., different subsets means the model chooses to use different information to make the final decision 2 .
Logic Traps in Evaluating Attribution Scores
The overview of our model is shown in Figure  7. To guarantee that only the subset information is included in the classifier, we discretely select the words and pass words instead of the hidden states of the extractor to the classifier. Since gradients do not flow through discrete samples, we resort to HardKuma (Bastings et al., 2019) to jointly train the model, which gives support to binary outcomes. HardKuma allows for setting the percentage of selected words and is proved more effective and stable than REINFORCE (Williams, 1992) in such scenarios. We set the selection ratio as 20% for SST-2 and 40% for MNLI because larger ratios will not cause further performance improvement. Finally, We get 85.6% accuracy on SST-2 and 66.2/65.5 % accuracy on MNLI-m/mm.
Logic Traps in Evaluating Attribution Scores
We use TextFooler (Jin et al., 2020) to generate adversarial samples. We use the same settings to Jin et al. (2020) to guarantee the semantical similarity of adversarial samples. The only difference is that we search for samples with minimal similarity in the selected subset instead of the model prediction. We guarantee that the model makes the same predictions, which is often used as the constraint for model reasoning in evaluation 3. We generate adversarial samples with 10% and 20% perturbation ratios.
Logic Traps in Evaluating Attribution Scores
We use F1-score to compute the similarity score between subsets and report the Macroaveraging F1-score of the whole development set.
Logic Traps in Evaluating Attribution Scores
10% 20% SST-2 0.32 0.18 MNLI-m/mm 0.43 / 0.52 0.37 / 0.43 Table 4: The similarity scores between selected subsets. Rato refers to the perturbation ratio used to generate adversarial samples.
Logic Traps in Evaluating Attribution Scores
A lower score is better, reflecting a larger difference in selected subsets. Note that since some words in original samples are replaced with their synonyms in adversarial samples, synonyms are seen as identical to their original words when evaluating. We evaluate all samples in the SST-2 development set and the first 1000 samples in MNLI-m/mm development sets. The results are shown in Table 4 As shown in Table 4, though semantically similar to the original samples and share the same model predictions, the adversarial samples can have subsets with low similarity to the original subset. Moreover, with a 10% perturbation ratio, 31.8% of samples in SST-2 have an adversarial subset with none word overlap with the original subset. This result increases to 50.5% with a 20% perturbation ratio. With no overlap between the two subsets, there is no way we can hypothesis the adversarial samples share similar model reasoning to the original samples.
Logic Traps in Evaluating Attribution Scores
Summarization Though evaluation 3 seems reasonable, sharing similar semantics and the same model predictions is a weak constraint for similar model reasoning. Thus the change in attribution scores may come from different model reasoning instead of the instability of attribution methods. Because of deep models' high sensitivity to adversarial samples, works resorting to evaluation 3 all get the same conclusion that existing attribution methods are fragile or unreliable. We argue we should find a more strict constraint for model reasoning first instead of ignoring logic trap 3 and disproving attribution methods recklessly.
Logic Traps in Evaluating Attribution Scores
replacing the target model.
Logic Traps in Evaluating Attribution Scores
Besides resorting to methods in evaluation 3, there are works (Jain and Wallace, 2019;Wang et al., 2020;Slack et al., 2020) disprove the reliability of attribution methods by replacing the target model which attribution methods should work on.
Logic Traps in Evaluating Attribution Scores
For example, Slack et al. (2020) trains an adversarial classifier e(x) to distinguish whether the inputs have been perturbed or not and then uses a different sub-model to process perturbed instances. Specifically, if we want to attack the LOO method, we can build a loo set from the original dataset and train e(x) in the following form:
Logic Traps in Evaluating Attribution Scores
This way, ψ(x), a model irrelevant to model predictions, is used when using LOO to calculate attribution scores, making generated attribution scores meaningless. Slack et al. (2020) assert that results of perturbation-based attribution methods such as LIME and SHAP (Lundberg and Lee, 2017) are easily attacked by their method. Similarly, Wang et al. (2020) add an extra model to the original model, which has uniform outputs but large gradients for some particular tokens such as 'CLS' in BERT. Since the extra model generates uniform outputs, it will not affect predictions of the original model. However, the extra model's gradients will add to the original model and thus can confuse gradient-based attribution methods.
Logic Traps in Evaluating Attribution Scores
The attack methods in Section 3.1 fool the attribution methods through designing a special structure and require attribution methods to be used in a black-box way. In this setting, the attribution methods are easily attacked and generate meaningless results. However, the question is: as a tool to help humans understand how deep models work, is it necessary to use attribution methods in a black-box way? Take the linear model as an example. The linear model is regarded as a white-box model, and humans don't need attribution methods to understand how it works. However, the understanding of a linear model is based on the analysis of its calculation process. Meanwhile, the deep model is regarded as a black-box model because its calculation process is too complicated to understand for humans, not because its calculation process is inaccessible. Thus, we believe there are no compelling reasons to require attribution methods to be used in a black-box way. The attacks in Wang et al. (2020); Slack et al. (2020) will fail when humans use attribution methods with knowing the model structures.
Logic Traps in Evaluating Attribution Scores
Since logic traps in existing evaluation methods can cause an inaccurate evaluation, we believe reducing the impact of these traps is the next question in the research field of post-hoc interpretations. In this section, we provide some thoughts for reducing the impact of logic trap 3:
Logic Traps in Evaluating Attribution Scores
The change in attribution scores may be because the model reasoning process is changed rather than the attribution method is unreliable.
Logic Traps in Evaluating Attribution Scores
To reduce the impact of this logic trap, we should try to guarantee the similarity in model reasoning when processing semantically similar inputs. In other words, we hope the target model used to test attribution methods more robustness to adversarial samples, which can be conducted through the following ways:
Logic Traps in Evaluating Attribution Scores
1 Enhancing the target model. The success of adversarial attacks on deep models motivates efforts to defend against such attacks. Thus, we can use these defense techniques, such as adversarial training (Tramèr et al., 2018) and randomization (Xie et al., 2018), to enhance the target model and make it more robustness to adversarial samples.
Logic Traps in Evaluating Attribution Scores
2 Excluding predictions with low confidence.
Logic Traps in Evaluating Attribution Scores
The deep model will give a prediction for a sample regardless of whether knowing how to deal with it. The randomness of reasoning increases with the uncertainty in model decisions (Bella et al., 2010). Thus, we can guarantee the stability of model reasoning by excluding low-confident predictions. For example, we can resorting to Confidence Calibration techniques (Guo et al., 2017;Seo et al., 2019), which calculate confidence interval for a predicted response.
Logic Traps in Evaluating Attribution Scores
The proposed logic traps in existing evaluation methods have been ignored for a long time and negatively affected the research field. Though strictly accurate evaluation metrics for evaluating attribution methods might be a "unicorn" which will likely never be found, we should not just ignore these logic traps and draw conclusions recklessly. With a clear statement and awareness of these logic traps, we should reduce the focus on improving performance under such unreliable evaluation systems and shift it to reducing the impact of proposed logic traps. Moreover, other aspects of the research field should give rise to more attention, such as the applications of attribution scores (denoising data, improving the model performance, etc.) and proposing new explanation forms.
Logic Traps in Evaluating Attribution Scores
In this section, we provide the experimental details of our experiments. Moreover, we will release our code and model within two months.
Logic Traps in Evaluating Attribution Scores
We merged dev-high and dev-middle sets as the development set. As shown in Figure 8, the document D, question Q, and one of the choices C are concatenated together as the input of model, and we replace the development set questions with empty strings in our experiment. 
Logic Traps in Evaluating Attribution Scores
We use the tokenizer of BERT to split the sentence into words in experiment 2. We modify 20% of words in the sentences in experiment 2. Since the word number in a sentence is not necessarily a multiple of five, we need to choose between rounding up or down. We use the same setting in code of HEDGE, i.e., rounding down. Specifically, we modify one word when word number is smaller than five.
Logic Traps in Evaluating Attribution Scores
Since HardKuma allows for setting the percentage of selected words, we first experiment with settings ranging from 10% to 100%. The results are shown in Figure 9. Under the premise of maintaining model performance, we choose the smallest setting (20% setting for SST-2 and 40% setting for MNLI). We use beam search to find adversarial samples and set the maximum reserved sample number to 100. 
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Interest has recently grown in interpretable NLP systems that can reveal how and why models make their predictions. But work in this direction has been conducted on different datasets with correspondingly different metrics, and the inherent subjectivity in defining what constitutes 'interpretability' has translated into researchers using different metrics to quantify performance. We aim to facilitate measurable progress on designing interpretable NLP models by releasing a standardized benchmark of datasets -augmented and repurposed
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Where do you find the most amount of leafs?
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In this movie, … Plots to take over the world. The acting is great! The soundtrack is run-of-the-mill, but the action more than makes up for it (a) Positive (b) Negative
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Article Patients for this trial were recruited … Compared with 0.9% saline, 120 mg of inhaled nebulized furosemide had no effect on breathlessness during exercise.  from pre-existing corpora, and spanning a range of NLP tasks -and associated metrics for measuring the quality of rationales. We refer to this as the Evaluating Rationales And Simple English Reasoning (ERASER) benchmark.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In curating and releasing ERASER we take inspiration from the stickiness of the GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) benchmarks for evaluating progress in natural language understanding tasks. These have enabled rapid progress on models for general language representation learning. We believe the still somewhat nascent subfield of interpretable NLP stands to similarly benefit from an analogous collection of standardized datasets/tasks and metrics.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
'Interpretability' is a broad topic with many possible realizations (Doshi-Velez and Kim, 2017;Lipton, 2016). In ERASER we focus specifically on rationales, i.e., snippets of text from a source doc-ument that support a particular categorization. All datasets contained in ERASER include such rationales, explicitly marked by annotators as supporting particular categorizations. By definition rationales should be sufficient to categorize documents, but they may not be comprehensive. Therefore, for some datasets we have collected comprehensive rationales, i.e., in which all evidence supporting a classification has been marked.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
How one measures the 'quality' of extracted rationales will invariably depend on their intended use. With this in mind, we propose a suite of metrics to evaluate rationales that might be appropriate for different scenarios. Broadly, this includes measures of agreement with human-provided rationales, and assessments of faithfulness. The latter aim to capture the extent to which rationales provided by a model in fact informed its predictions.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
While we propose metrics that we think are reasonable, we view the problem of designing metrics for evaluating rationales -especially for capturing faithfulness -as a topic for further research that we hope that ERASER will help facilitate. We plan to revisit the metrics proposed here in future iterations of the benchmark, ideally with input from the community. Notably, while we provide a 'leaderboard', this is perhaps better viewed as a 'results board'; we do not privilege any one particular metric. Instead, we hope that ERASER permits comparison between models that provide rationales with respect to different criteria of interest.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We provide baseline models and report their performance across the corpora in ERASER. While implementing and initially evaluating these baselines, we found that no single 'off-the-shelf' architecture was readily adaptable to datasets with very different average input lengths and associated rationale snippets. This suggests a need for the development of new models capable of consuming potentially lengthy input documents and adaptively providing rationales at the level of granularity appropriate for a given task. ERASER provides a resource to develop such models, as it comprises datasets with a wide range of input text and rationale lengths (Section 4).
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In sum, we introduce the ERASER benchmark (www.eraserbenchmark.com), a unified set of diverse NLP datasets (repurposed from existing corpora, including sentiment analysis, Natural Language Inference, and Question Answering tasks, among others) in a standardized format featuring human rationales for decisions, along with the starter code and tools, baseline models, and standardized metrics for rationales.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In this section we discuss properties that might be desirable in rationales, and the metrics we propose to quantify these (for evaluation). We attempt to operationalize these criteria formally in Section 5.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
As one simple metric, we can assess the degree to which the rationales extracted by a model agree with those highlighted by human annotators. To measure exact and partial match, we propose adopting metrics from named entity recognition (NER) and object detection. In addition, we consider more granular ranking metrics that account for the individual weights assigned to tokens (when models assign such token-level scores, that is).
ERASER: A Benchmark to Evaluate Rationalized NLP Models
One distinction to make when evaluating rationales is the degree to which explanation for predictions is desired. In some cases it may be important that rationales tell us why a model made the prediction that it did, i.e., that rationales are faithful. In other settings, we may be satisfied with "plausible" rationales, even if these are not faithful.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Another key consideration is whether one wants rationales that are comprehensive, rather than simply sufficient. A comprehensive set of rationales comprises all snippets that support a given label. Put another way, if we remove a comprehensive set of rationales from an instance, there should be no way to categorize it (Yu et al., 2019). ERASER permits evaluation of comprehensiveness by including exhaustive annotated rationales that we have collected for some of datasets in the benchmark.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In this section we describe the datasets that comprise the proposed rationales benchmark. All datasets constitute predictive tasks for which we distribute both reference labels and spans marked by humans, in a standardized format. For some of the datasets we have acquired comprehensive rationales from humans for a subset of instances. This permits evaluation of model recall, with respect to extracted rationales.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We distribute train, validation, and test sets for all corpora (see Appendix A for processing details). We ensure that these sets comprise disjoint sets of source documents to avoid contamination. 2 We have made the decision to distribute the test sets publicly, 3 in part because we do not view the 'correct' metrics to use as settled. We plan to acquire additional human annotations on held-out portions of some of the included corpora so as to offer hidden test set evaluation opportunities in the future.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Evidence inference (Lehman et al., 2019). This is a dataset of full-text articles describing the conduct and results of randomized controlled trials (RCTs). The task is to infer whether a given intervention is reported to either significantly increase, significantly decrease, or have no significant effect on a specified outcome, as compared to a comparator of interest. A justifying rationale extracted from the text should be provided to support the inference. As the original annotations are not necessarily exhaustive, we collect exhaustive annotations on a subset of the test data 4 .
ERASER: A Benchmark to Evaluate Rationalized NLP Models
BoolQ (Clark et al., 2019). This corpus consists of passages selected from Wikipedia, and yes/no questions generated from these passages. As the original Wikipedia article versions used were not maintained, we have made a best-effort attempt to recover these, and then find within them the passages answering the corresponding questions. For public release, we acquired comprehensive annotations on a subset of documents in our test set 4 .
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Movie Reviews (Zaidan and Eisner, 2008). One of the original datasets providing extractive rationales, the movies dataset has positive or negative sentiment labels on movie reviews. As the included rationale annotations are not necessarily comprehensive (i.e., annotators were not asked to mark all  text supporting a label), we collect a comprehensive evaluation set on the final fold of the original dataset (Pang and Lee, 2004) 4 .
ERASER: A Benchmark to Evaluate Rationalized NLP Models
FEVER (Thorne et al., 2018). FEVER 1.0 (short for Fact Extraction and VERification) is a factchecking dataset. The task is to verify claims from textual sources. In particular, each claim is to be classified as supported, refuted or not enough information with reference to a collection of potentially relevant source texts. We restrict this dataset to supported or refuted.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
MultiRC (Khashabi et al., 2018). This is a reading comprehension dataset composed of questions with multiple correct answers that by construction depend on information from multiple sentences. In MultiRC, each Rationale is associated with a question, while answers are independent of one another. We convert each rationale/question/answer triplet into an instance within our dataset. Each answer candidate then has a label of True or False. In ERASER, models are evaluated both for their 'downstream' performance (i.e., performance on the actual classification task) and with respect to the rationales that they extract. For the former we rely on the established metrics for the respective tasks. Here we describe the metrics we propose to evaluate the quality of extracted rationales. We do not claim that these are necessarily the best metrics for evaluating rationales, but they are reasonable starting measures. We hope the release of ERASER will spur additional research into how best to measure the quality of model explanations in the context of NLP.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
The simplest means of evaluating rationales extracted by models is to measure how well they agree with those marked by humans. To this end we propose two classes of metrics: those based on exact matches, and ranking metrics that provide a measure of the model's ability to discriminate between evidence and non-evidence tokens (appropriate for models that provide soft scores for tokens). For the former, we borrow from Named Entity Recognition (NER); we effectively measure the overlap between spans extracted and marked. Specifically, given an extracted set of l rationales {r 1 , ..., r l } extracted for instance i, we compute precision, recall, and F1 with respect to m human rationales {h 1 , ..., h m }.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Exact match is a particularly harsh metric in that it may not reflect subjective rationale quality; consider that an extra token destroys the match but not (usually) the meaning. We therefore consider softer variants. Intersection-Over-Union (IOU), borrowed from computer vision (Everingham et al., 2010), permits credit assignment in the case of partial matches. We define IOU on a token level: for two spans x, y, it is the size of the overlap of the tokens covered by the spans divided by the size of the union. We count a prediction as a match if it overlaps with any of the ground truth rationales by more than some threshold (0.5 for this work). We compute true positives from these matches; other measures (false positives, false negatives) are computed normally, and yield a more forgiving precision, recall, and F-measure.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We provide two additional relaxations of the exact match metric. First, a token-level precision, recall, and F1 allow for a broader sense of model coverage, although these ignore contiguousness, which is likely a desirable property of rationales. Systems may also provide a sentence-level decision as a second relaxed scoring metric. In general we consider token and span-level metrics superior to sentence metrics as they are more granular, but some datasets have meaningful sentence level annotations. 5 Our second class of metrics considers rankings. This rewards models for assigning relatively highscores to marked tokens. In particular, we take the Area Under the Precision-Recall curve (AUPRC) constructed by sweeping a threshold over token scores.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In general, the rationales we have for tasks are sufficient to make judgments, but not necessarily comprehensive. However, for some datasets we have explicitly collected comprehensive rationales for at least a subset of the test set. Therefore, on these datasets recall evaluates comprehensiveness directly (it does so only noisily on other datasets). We highlight which corpora contain comprehensive rationales in the test set in Table 4.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Above we proposed simple metrics for agreement with human-provided rationales. But as discussed above, a model may provide rationales that are plausible (and agree with those marked by humans) but that it did not in fact rely on to come to its disposition. In some scenarios this may be acceptable, but in many settings one may want rationales that actually explain model predictions, i.e., rationales extracted for an instance in this case ought to have meaningfully influenced its prediction for the same. We refer to these as faithful rationales.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
How best to measure the faithfulness of rationales is an open question. In this first version of ERASER we propose a few straightforward metrics motivated by prior work (Zaidan et al., 2007;Yu et al., 2019). In particular, following Yu et al. (2019) we define metrics intended to capture the comprehensiveness and sufficiency of rationales, respectively. The former should capture whether all features needed to come to a prediction were selected, and the latter should tell us whether the extracted rationales contain enough signal to come to a disposition.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Comprehensiveness. To calculate rationale comprehensiveness we create contrast examples (Zaidan et al., 2007) by taking an input instance x i with rationales r i and erasing from the former all tokens found in the latter. That is, we construct a contrast example for x i ,x i , which is x i with the rationales removed. Assuming a simple classification setting, letp ij be the original prediction provided by a model m for the predicted class j:p ij = m(x i ) j . Then we consider the predicted probability from the model for the same class once the supporting rationales are stripped:p ij = m(x i ). Intuitively, the model ought to be less confident in its prediction once rationales are removed from x i . We can measure this as:
ERASER: A Benchmark to Evaluate Rationalized NLP Models
If this is high, this implies that the rationales were indeed influential in the prediction; if it is low, then this suggests that they were not. A negative value here means that the model became more confident in its prediction after the rationales were removed; this would seem quite counter-intuitive if the rationales were indeed the reason for its prediction in the first place.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Sufficiency. The second metric for measuring the faithfulness of rationales that we use is intended to capture the degree to which the snippets within the extracted rationales are adequate for a model to make a prediction. Denote byp ij the predicted probability of class j using only rationales r i . Then:
ERASER: A Benchmark to Evaluate Rationalized NLP Models
These metrics are illustrated in Figure 2. As defined, the above measures have assumed discrete rationales r i . We would like also to evaluate the faithfulness of continuous importance scores assigned to tokens by models. Here we adopt a simple approach for this. We convert soft scores over features s i provided by a model into discrete rationales r i by taking the top−k d values, where k d is a threshold for dataset d. We set k d to the average rationale length provided by humans for dataset d (see Table 4). Intuitively, this says: How much does the model prediction change if we remove a number of tokens equal to what humans use (on average for this dataset) in order of the importance scores assigned to these by the model. Once we have discretized the soft scores into rationales in this way, we compute the faithfulness scores as per Equations 1 and 2. This approach is conceptually simple. It is also computationally cheap to evaluate, in contrast to measures that require per-token measurements, e.g., importance score correlations with 'leave-one-out' scores (Jain and Wallace, 2019), or counting how many 'important' tokens need to be erased before a prediction flips (Serrano and Smith, 2019). However, the necessity of discretizing continuous scores forces us to rely on the rather ad-hoc application of threshold k d . We believe that picking this based on human rationale annotations per dataset is reasonable, but acknowledge that alternative choice of threshold may yield quite different results for a given model and rationale set. It may be better to construct curves of this measure across varying k d and compare these, but this is both subtle (such curves will not necessarily be monotonic) and computationally intensive.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Ultimately, we hope that ERASER inspires additional research into designing faithfulness metrics for rationales. We plan to incorporate additional such metrics into future versions of the benchmark, if appropriate.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Our focus in this work is primarily on the ERASER benchmark itself, rather than on any particular model(s). However, to establish initial empirical results that might provide a starting point for future work, we evaluate several baseline models across the corpora in ERASER. 6 We broadly class these into models that assign 'soft' (continuous) scores to tokens, and those that perform a 'hard' (discrete) selection over inputs. We additionally consider models specifically designed to select individual tokens (and very short sequences) as rationales, as compared to longer snippets.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We describe these models in the following subsections. All of our implementations are available in the ERASER repository. Note that we do not aim to provide, by any means, a comprehensive suite of models: rather, our aim is to establish a reasonable starting point for additional work on such models.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
All of the datasets in ERASER have a similar structure: inputs, rationales, labels. But they differ considerably in length (Table 4), both of documents and corresponding rationales. We found that p(Forest|x i )
ERASER: A Benchmark to Evaluate Rationalized NLP Models
A U 0 I w n U 9 w Q 0 v M H V 2 G / c g 1 Q 8 D O 5 w F I H r 0 1 7 A u 5 x R 1 F I 7 f + j 0 K S Z R W k w c h C E m 1 6 E E h W n 6 M G z z k 3 a + Y J W s C c x F Y s 9 I g c x Q b e e / n E 7 I Y h 8 C Z I I q 1 b K t C N 2 E S u R M Q J p z Y g U R Z Q P a g 5 a m A f V B u c n k i 9 Q 8 1 k r H 7 I Z S V 4 D m R P 0 9 k V B f q Z H v 6 U 6 f Y l / N e 2 P x P 6 8 V Y / f C T X g Q x Q g B m y 7 q x s L E 0 B x H Y n a 4 B I Z i p A l l k u t b T d a n k j L U w e V 0 C P b 8 y 4 u k X i 7 Z p 6 X y 7 V m h c j m L I 0 s O y B E p E p u c k w q 5 I V V S I 4 w 8 k m f y S t 6 M J + P F e D c + p q 0 Z Y z a z T / 7 A + P w B u O K Z W Q = = < / l a t e x i t > Where do you find the most amount of leafs?  this motivated use of different models for datasets, appropriate to their sizes and rationale granularities. In our case this was in fact necessitated by computational constraints, as we were unable to run larger models on lengthier documents such as those within Evidence Inference. We hope that this benchmark motivates design of models that provide rationales that can flexibly adapt to varying input lengths and expected rationale granularities. Indeed, only with such models can we perform comparisons across datasets.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Models that perform hard selection may be viewed as comprising two independent modules: an encoder which is responsible for extracting snippets of inputs, and a decoder that makes a prediction based only on the text provided by the encoder. We consider two variants of such models. Lei et al. (2016). In this model, the encoder induces a binary mask over inputs x, z. The decoder consumes the attributes of x indicated by z to make a predictionŷ. The components are typically trained jointly. This end-to-end training is complicated by the use of (non-differentiable) hard attention, i.e., the binary mask z, which means it is not possible to train the model using variants of gradient descent. Instead, Lei et al. (2016) propose using REINFORCE (Williams, 1992) style estimation, minimizing the loss over expected binary vectors z yielded from the encoder.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
One of the advantages of this approach is that it need not have access to marked rationales; it can learn to rationalize on the basis of instance labels alone. However, given that here we do have access to rationales in the training data, we experiment with a variant in which we train the encoder explicitly using rationale-level annotations.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In our implementation of Lei et al. (2016), we drop in two independent BERT (Devlin et al., 2018) base modules with bidirectional LSTM (Hochreiter and Schmidhuber, 1997) on top to induce contextualized representations of tokens for the encoder and decoder (the decoder, in addition, uses additive attention to collapse the LSTM hidden representations to a single vector), respectively. The encoder generates a scalar (denoting the probability of selecting that token) for each LSTM hidden state using a feedfoward layer and sigmoid. In the model where we do use human rationales during training, we minimize binary cross entropy between our sigmoid output and the ground truth rationale. Thus our final loss function is composed of decoder classification loss, reinforce estimator loss (details can be found in Lei et al. (2016)) and if used, a rationale supervision loss.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Pipeline models. These are simple models in which we first train the encoder to extract rationales, and then train the decoder to perform prediction using only rationales. No parameters are shared between the two models. Realizing this type of approach is possible only when one has access to direct rationale supervision in order to train the encoder (which in general we assume in ERASER).
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Here we first consider a simple pipeline that first segments inputs into sentences. It passes these, one at a time, through a Gated Recurrent Unit (GRU)  to yield hidden representations that we compose via an attentive decoding layer . This aggregate representation is then passed to a classification module which predicts whether the corresponding sentence is a rationale (or not). A second model, using effectively the same architecture but parameterized independently, consumes the outputs (rationales) from the first to make predictions. This simple model is described at length in prior work (Lehman et al., 2019). We further consider a 'BERT-to-BERT' pipeline, where we replace each stage with a BERT module for prediction (Devlin et al., 2018).
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In all pipeline models, we train each stage independently. The rationale identification stage is trained using approximate sentence boundaries from our source annotations, with randomly sampled negative examples at each epoch. The classification stage uses the same positive rationales as the identification stage, a kind of teacher forcing. See Appendix C for more detail.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
A subset of datasets in ERASER contain tokenlevel annotations, i.e., in these cases individual words and/or comparatively short sequences of words are marked as supporting classification decisions. These are: MultiRC, Movies, e-SNLI, and CoS-E. For these datasets we consider a model that passes tokens through BERT (Devlin et al., 2018) to induce contextualized representations that are then passed to a bi-directional LSTM (Hochreiter and Schmidhuber, 1997). The hidden representations from the LSTM are collapsed into a single vector using additive attention  and finally through a linear layer followed by a sigmoid to yield (per-token) relevance predictions. We use the LSTM layer in part to bypass the 512 word limit imposed by BERT; when we exceed this length, we effectively start encoding a 'new' sequence (setting the positional index to 0) via BERT. The hope is that the LSTM layer learns to compensate for this. We have not yet trained this model on larger corpora due to computational constraints. For now we instead use a similar setup as above for Evidence Inference, BoolQ, and FEVER, except we swap in GloVe 300-d embeddings (Pennington et al., 2014) in place of BERT representations for tokens.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
For these models we consider input gradients (with respect to output) and attention induced over contextualized representations as 'soft' scores.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Performance IOU F1 Token F1
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Evidence Inference (Lehman et al., 2019) 0  Here we present initial results for the baseline models discussed in Section 6, with respect to the metrics proposed in Section 5. We present results in two parts, reflecting the two classes of rationales discussed above: 'hard' approaches that perform discrete selection of snippets and 'soft' methods that assign continuous importance scores to tokens.   First, in Table 3 we evaluate models that perform discrete selection of rationales. We view these models as faithful by design, because by construction we know what snippets of text the decoder used to make a prediction. 7 Therefore, for these methods we report only metrics that measure agreement with human annotations.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
Due to computational constraints, we are currently unable to run our BERT-based implementation of Lei et al. (2016) over larger corpora. Conversely, Lehman et al. (2019) assumes a setting in which rationale are sentences, and so is not appropriate for datasets in which rationales tend to comprise only very short spans. Again, in our view this highlights the need for models that can rationalize at varying levels of granularity, depending on what is appropriate.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We observe that for the "rationalizing" model of Lei et al. (2016), exploiting rationale-level supervision generally improves agreement with humanprovided rationales, which is consistent with prior work (Zhang et al., 2016;Strout et al., 2019). Here, Lei et al. (2016) consistently outperform the simple pipeline model from Lehman et al. (2019). Furthermore, Lei et al. (2016) outperforms the 'BERT-to-BERT' pipeline on the comparable datasets for the final classification tasks. This may be an artifact of the amount of text each model can select: 'BERTto-BERT' is limited to sentences, while Lei et al. (2016) can select any subset of the text.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
In Table 4 we report metrics for models that assign soft (continuous) importance scores to individual tokens. For these models we again measure downstream (task) performance (F1 or accuracy, as appropriate). Here the models are actually the same, and so downstream performance is equivalent. To assess the quality of token scores with respect to human annotations, we report the Area Under the Precision Recall Curve (AUPRC). Finally, as these scoring functions assign only soft scores to inputs (and may still use all inputs to come to a particular prediction), we report the metrics intended to measure faithfulness defined above: comprehensiveness and sufficiency. Here we observe that the simple gradient attribution yields consistently more 'faithful' rationales with respect to comprehensiveness, and in a slight majority of cases also with respect to sufficiency. Interestingly, however, attention weights yield better AUPRCs. 7 Note that this further assumes that the encoder and decoder do not share parameters.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We view these as preliminary results and intend to implement and evaluate additional baselines in the near future. Critically, we see a need for establishing the performance of a single architecture across ERASER, which comprises datasets of very different size, and featuring rationales at differing granularities.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We have described a new publicly available Evaluating Rationales And Simple English Reasoning (ERASER) benchmark. This comprises seven datasets, all of which have both instance level labels and corresponding supporting snippets ('rationales') marked by human annotators. We have augmented many of these datasets with additional annotations, and converted them into a standard format comprising inputs, rationales, and outputs. ERASER is intended to facilitate progress on explainable models for NLP.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We have proposed several metrics intended to measure the quality of rationales extracted by models, both in terms of agreement with human annotations, and in terms of 'faithfulness' with respect to comprehensiveness and sufficiency. We believe these metrics provide reasonable means of comparison of specific aspects of interpretability. However, we view the problem of measuring faithfulness, in particular, a topic ripe for additional research; we hope that ERASER facilitates this.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
More generally, our hope is that ERASER facilitates progress on designing and comparing relative strengths and weaknesses of interpretable NLP models across a variety of tasks and datasets. We aim to continually update this benchmark and the corresponding metrics that it defines. In contrast to most benchmarks, we are not privileging any one measure of performance. Our view is that for interpretability, different models may excel at different things, and our aim for ERASER is to facilitate meaningful contrastive comparisons that highlight which models excel with respect to particular metrics of interest (e.g., certain models may provide superior faithfulness, though with lower predictive performance). We host a leaderboard, but allow for sorting with respect to any metric of interest.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
The ERASER datasets, code for working with the data and performing evaluations, and our baseline model implementations are all available at: www.eraserbenchmark.com, which we will be continuously updating. C Hyperparameter and training details C.1 (Lei et al., 2016) 
ERASER: A Benchmark to Evaluate Rationalized NLP Models
For these models, we set the sparsity rate at 0.01 and we set the contiguity loss weight to 2 times sparsity rate (following the original paper). We used bert-base-uncased (Wolf et al., 2019) as token embedder and Bidirectional LSTM with 128 dimensional hidden state in each direction. A dropout (Srivastava et al., 2014) rate of 0.2 was used before feeding the hidden representations to attention layer in decoder and linear layer in encoder. One layer MLP with 128 dimensional hidden state and ReLU activation was used to compute the decoder output distribution.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
A learning rate of 2e-5 with Adam (Kingma and Ba, 2014) optimizer was used for all models and we only fine-tuned top two layers of BERT encoder. Th models were trained for 20 epochs and early stopping with patience of 5 epochs was used. The best model was selected on validation set using the final task performance metric.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
The input for the above model was encoded in form of [CLS] document [SEP] query [SEP].
ERASER: A Benchmark to Evaluate Rationalized NLP Models
This model was implemented using AllenNLP library (Gardner et al., 2017).
ERASER: A Benchmark to Evaluate Rationalized NLP Models
This model is essentially the same as decoder in previous section. The BERT-LSTM uses the same hyperparameter and GloVe-LSTM is trained with a learning rate of 1e-2.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
With the exception of the Evidence Inference dataset, these models were trained using the GLoVe (Pennington et al., 2014) 200 dimension word vectors, and Evidence Inference using the (Pyysalo et al., 2013) PubMed word vectors. We use Adam (Kingma and Ba, 2014) with a learning rate of 1e-3, Dropout (Srivastava et al., 2014) of 0.05 at each layer (embedding, GRU, attention layer) of the model, for 50 epochs with a patience of 10. We monitor validation loss, and keep the best model on the validation set.
ERASER: A Benchmark to Evaluate Rationalized NLP Models
We primarily used the bert-base-uncased model for both portions of the identification and classification pipeline, with the sole exception being Evidence Inference with SciBERT (Beltagy et al., 2019).
Is Attention Interpretable?
Interpretability is a pressing concern for many current NLP models. As they become increasingly complex and learn decision-making functions from data, ensuring our ability to understand why a particular decision occurred is critical.
Is Attention Interpretable?
Part of that development has been the incorporation of attention mechanisms (Bahdanau et al., 2015) into models for a variety of tasks. For many different problems-to name a few, machine translation (Luong et al., 2015), syntactic parsing (Vinyals et al., 2015), reading comprehension (Hermann et al., 2015), and language modeling (Liu and Lapata, 2018)-incorporating attention mechanisms into models has proven beneficial for performance. While there are many variants of attention (Vaswani et al., 2017), each for-mulation consists of the same high-level goal: calculating nonnegative weights for each input component (e.g., word) that together sum to 1, multiplying those weights by their corresponding representations, and summing the resulting vectors into a single fixed-length representation.
Is Attention Interpretable?
Since attention calculates a distribution over inputs, prior work has used attention as a tool for interpretation of model decisions (Wang et al., 2016;Lee et al., 2017;Lin et al., 2017;Ghaeini et al., 2018). The existence of so much work on visualizing attention weights is a testament to attention's popularity in this regard; to name just a few examples of these weights being examined to understand a model, recent work has focused on goals from explaining and debugging the current system's decision (Lee et al., 2017;Ding et al., 2017) to distilling important traits of a dataset (Yang et al., 2017;Habernal et al., 2018).
Is Attention Interpretable?
Despite this, existing work on interpretability is only beginning to assess what computed attention weights actually communicate. In an independent and contemporaneous study, Jain and Wallace (2019) explore whether attention mechanisms can identify the relative importance of inputs to the full model, finding them to be highly inconsistent predictors. In this work, we apply a different analysis based on intermediate representation erasure to assess whether attention weights can instead be relied upon to explain the relative importance of the inputs to the attention layer itself. We find similar cause for concern: attention weights are only noisy predictors of even intermediate components' importance, and should not be treated as justification for a decision.
Is Attention Interpretable?
We focus on five-and ten-class text classification models incorporating attention, as explaining the reasons for text classification has been a particular area of interest for recent work in interpretability (Yang et al., 2016;Ribeiro et al., 2016;Lei et al., 2016;Feng et al., 2018).
Is Attention Interpretable?
In order for a model to be interpretable, it must not only suggest explanations that make sense to people, but also ensure that those explanations accurately represent the true reasons for the model's decision. Note that this type of analysis does not rely on the true labels of the data; if a model produces an incorrect output, but a faithful explanation for which factors were important in that calculation, we still consider it interpretable.
Is Attention Interpretable?
We take the implied explanation provided by visualizing attention weights to be a ranking of importance of the attention layer's input representations, which we denote I: if the attention allocated to item i ∈ I is higher than that allocated to item j ∈ I, then i is presumed "more important" than j to the model's output. In this work, we focus on whether the attention weights' suggested importance ranking of I faithfully describes why the model produced its output, echoing existing work on explanation brittleness for other model components (Ghorbani et al., 2017).
Is Attention Interpretable?
We are interested in the impact of some contextualized inputs to an attention layer, I ⊂ I, on the model's output. To examine the importance of I , we run the classification layer of the model twice ( Figure 1): once without any modification, and once after renormalizing the attention distribution with I 's attention weights zeroed out, sim-ilar to other erasure-based work (Li et al., 2016;Feng et al., 2018). We then observe the resulting effects on the model's output. We erase at the attention layer to isolate the effects of the attention layer from the encoder preceding it. Our reasoning behind renormalizing is to keep the output document representation from artificially shrinking closer to 0 in a way never encountered during training, which could make subsequent measurements unrepresentative of the model's behavior in spaces to which it does map inputs.
Is Attention Interpretable?
One point worth noting is the facet of interpretability that our tests are designed to capture. By examining only how well attention represents the importance of intermediate quantities, which may themselves already have changed uninterpretably from the model's inputs, we are testing for a relatively low level of interpretability. So far, other work looking at attention has examined whether attention suffices as a holistic explanation for a model's decision (Jain and Wallace, 2019), which is a higher bar. We instead focus on the lowest standard of interpretability that attention might be expected to meet, ignoring prior model layers.
Is Attention Interpretable?
We denote the output distributions (over labels) as p (the original) and q I (where we erase attention for I ). The question now becomes how to operationalize "importance" given p and q I . There are many quantities that could arguably capture information about importance. We focus on two: the Jensen-Shannon (JS) divergence between output distributions p and q I , and whether the argmaxes of p and q I differ, indicating a decision flip.
Is Attention Interpretable?
We investigate four model architectures on a topic classification dataset (Yahoo Answers;Zhang et al., 2015) and on three review ratings datasets: IMDB (Diao et al., 2014), 2 Yelp 2017, 3 and Amazon (Zhang et al., 2015). Statistics for each dataset are listed in Table 1.
Is Attention Interpretable?
Our model architectures are inspired by the hierarchical attention network (HAN;Yang et al., 2016), a text classification model with two layers of attention, first to the word tokens in each sentence and then to the resulting sentence representations. The layer that classifies the document representation is linear with a softmax at the end.
Is Attention Interpretable?
We conduct our tests on the softmax formula- tion of attention, 4 which is used by most models, including the HAN. Specifically, we use the additive formulation originally defined in Bahdanau et al. (2015). Given attention layer 's learned parameters, element i of a sequence, and its encoded representation h i , the attention weight α i is computed using 's learned context vector c as follows:
Is Attention Interpretable?
We evaluate on the original HAN architecture, but we also vary that architecture in two key ways:
Is Attention Interpretable?
1. Number of attention layers: besides exploring models with a final layer of attention over sentence representations (which we denote with a "HAN" prefix), we also train "flat" attention networks with only one layer of attention over all contextualized word tokens (which we denote with a "FLAN" prefix). In either case, though, we only run tests on models' final layer of attention.
Is Attention Interpretable?
2. Reach of encoder contextualization: The original HAN uses recurrent encoders to contextualize input tokens prior to an attention layer (specifically, bidirectional GRUs running over the full sequence). Aside from biRNNs, we also experiment with models that instead contextualize word vectors by convolutions on only a token's close neighbors, inspired by Kim (2014). See Figure 2 for a diagram of the FLAN architecture using a convolutional encoder. We denote this variant of an architecture with a "conv" suffix. Finally, we also test models that are trained with no contextualizing encoder at all; we denote these with a "noenc" suffix.
Is Attention Interpretable?
The classification accuracy of each of our trained models is listed in Table 3 in the appendix, along with training details for the different models.
Is Attention Interpretable?
As a starting point for our tests, we investigate the relative importance of attention weights when only one weight is removed. Let i * ∈ I be the component with the highest attention and let α i * be its attention. We compare i * 's importance to some other attended item's importance in two ways.
Is Attention Interpretable?
We wish to compare how i * 's impact on the model's output distribution compares to the impact corresponding to a random attended item r drawn uniformly from I. Our first approach to this will be to calculate two JS divergences-one being the JS divergence of the model's original output distribution from its output distribution after removing only i * , and the other after removing only r-and compare them to each other. We subtract the output JS divergence after removing r from the output JS divergence after removing i * :
Is Attention Interpretable?
We plot this quantity against the difference ∆α = α i * −α r in Figure 3. We show results on the HANrnn, as the trends for the other models are very similar; see Figures 7-8 and the tables in Figure 9 in the Appendix for full results.   Table 1.
Is Attention Interpretable?
Intuitively, if i * is truly the most important, then we would expect Eq. 1 to be positive, and that is what we find the vast majority of the time. In addition, examining Figure 3, we see that nearly all negative ∆JS values are close to 0. By binning occurrences of negative ∆JS values by the difference between α i * and α r in Figure 4, we also see that in the cases where i * had a smaller effect, the gap between i * 's attention and r's tends to be small. This is encouraging, indicating that in these cases, i * and r are nearly "tied" in attention.
Is Attention Interpretable?
However, the picture of attention's interpretability grows somewhat more murky when we begin to consider the magnitudes of positive ∆JS values in Figure 3. We notice across datasets that even for quite large differences in attention weights like 0.4, many of the positive ∆JS are still quite close to zero. Although we do finally see an upward swing in ∆JS values once ∆α gets even larger, indicating only one very high attention weight in the distribution, this still leaves many open questions about exactly how much difference in impact i * and r can typically be expected to have.  Since attention weights are often interpreted as an explanation for a model's argmax decision, our second test looks at another, more immediately visible change in model outputs: decision flips. For clarity, we limit our discussion to results for the HANrnns, which reflect the same patterns observed for the other architectures. (Results for all other models are in Appendix A.2.) Table 2 shows, for each dataset, a contingency table for the two binary random variables (i) does zeroing α i * (and renormalizing) result in a decision flip? and (ii) does doing the same for a different randomly chosen weight α r result in a decision flip? To assess the comparative importance of i * and r, we consider when exactly one erasure changes the decision (off-diagonal cells). For attention to be interpretable, the blue, upper-right values (i * , not r, flips a decision) should be much larger than the orange, lower-left values (r, not i * , flips a decision), which should be close to zero. 5 Although for some datasets in Table 2, the "orange" values are non-negligible, we mostly see that their fraction of total off-diagonal values mirrors the fraction of negative occurrences of Eq. 1 in Figure 4. However, it's somewhat startling that in the vast majority of cases, erasing i * does not change the decision ("no" row of each table). This is likely explained in part by the signal pertinent to the classification being distributed across a document (e.g., a "Sports" question in the Yahoo Answers dataset could signal "sports" in a few sentences, any one of which suffices to correctly categorize it). However, given that these results are for the HAN models, which typically compute attention over ten or fewer sentences, this is surprising.
Is Attention Interpretable?
Altogether, examining importance from a single-weight angle paints a tentatively positive picture of attention's interpretability, but also raises several questions about the many cases where the difference in impacts between i * and r is almost identical (i.e., ∆JS values close to 0 or the many cases where neither i * nor r cause a decision flip). To answer these questions, we require tests with a broader scope.
Is Attention Interpretable?
Often, we care about determining the collective importance of a set of components I . To address that aspect of attention's interpretability and close gaps left by single-weight tests, we introduce tests to determine how multiple attention weights perform together as importance predictors.
Is Attention Interpretable?
For a hypothesized ranking of importance, such as that implied by attention weights, we would expect the items at the top of that ranking to function as a concise explanation for the model's decision. The less concise these explanations get, and the farther down the ranking that the items truly driving the model's decision fall, the less likely it becomes for that ranking to truly describe importance. In other words, we expect that the top items in a truly useful ranking of importance would comprise a minimal necessary set of information for making the model's decision. The idea of a minimal set of inputs necessary to uphold a decision is not new; Li et al. (2016) use reinforcement learning to attempt to construct such a minimal set of words, Lei et al. (2016) train an encoder to constrain the input prior to classification, and much of the work that has been done on extractive summarization takes this concept as a starting point (Lin and Bilmes, 2011). However, such work has focused on approximating minimal sets, instead of evaluating the ability of other importance-determining "shortcuts" (such as attention weight orderings) to identify them. Nguyen (2018) leveraged the idea of minimal sets in a much more similar way to our work, comparing different input importance orderings.
Is Attention Interpretable?
Concretely, to assess the validity of an importance ranking method (e.g., attention), we begin erasing representations from the top of the ranking downward until the model's decision changes. Ideally, we would then enumerate all possible subsets of that instance's components, observe whether the model's decision changed in response to removing each subset, and then report whether the size of the minimal decision-flipping subset was equal to the number of items that had needed to be removed to achieve a decision flip by following the ranking. However, the exponential number of subsets for any given instance's sequence of components (word or sentence representations, in our case) makes such a strategy computationally prohibitive, and so we adopt a different approach.
Is Attention Interpretable?
Instead, in addition to our hypothesized importance ranking (attention weights), we consider alternative rankings of importance; if, using those, we repeatedly discover cases where removing a smaller subset of items would have sufficed to change the decision, this signals that our candidate ranking is a poor indicator of importance.
Is Attention Interpretable?
Exhaustively searching the space of component subsets would be far too time-consuming in practice, so we introduce three other ranking schemes.
Is Attention Interpretable?
The first is to randomly rank importance. We expect that this ranking will perform quite poorly, but it provides a point of comparison by which to validate that ranking by descending attention weights is at least somewhat informative.
Is Attention Interpretable?
The second ranking scheme, inspired by Li et al. (2015) and Feng et al. (2018), is to order the attention weights by the gradient of the decision function with respect to each calculated attention weight, in descending order. Since each of the datasets on which we evaluate has either five or ten output classes, we take the decision function given a real-valued model output vector to be
Is Attention Interpretable?
Unlike the last two proposed rankings, our third ranking scheme uses attention weights, but supplements them with information about the gradient. For this ranking, we multiply each of our calculated gradients from our previous proposed ranking scheme by their corresponding attention weight magnitude. Under this ordering, attended items that have both a high attention weight and a high calculated gradient with respect to their attention weight will be ranked most important.
Is Attention Interpretable?
We introduce these last two rankings as an attempt to discover smaller sets not produced by the attention weight ranking. Note, however, that we still do not take either as a gold-standard indicator of importance to the model, as with the gradient in Ross et al. (2017) and Melis and Jaakkola (2018), but merely as an alternative ordering method. The "gold standard" in our case would be the minimal set of attention weights to zero out for the decision to change, which none of our ordering methods will necessarily find for a particular instance.
Is Attention Interpretable?
In cases where removing all but one input to the attention layer still does not produce a decision flip, we finish the process of removing components by removing the final representation and replacing the output of the attention layer with an arbitrary vector; we use the zero vector for our tests. Even so, since every real-valued vector output by the attention layer is mapped to an output distribution, removing this final item will still not change the classification decision for instances that the model happened to originally map to that same class. We exclude such instances for which the decision never changed from all subsequent analyses.
Is Attention Interpretable?
We also set aside any test instances with a sequence length of 1 for their final attention layer, as there is only one possible ordering for such cases.
Is Attention Interpretable?
Examining our results in Figure 5, we immediately see that ranking importance by descending attention weights is not optimal for our models with encoders. While removing intermediate representations in decreasing order by attention weights often leads to a decision flip faster than a random ranking, it also clearly falls short of matching (or even approaching) the decision-flipping efficiency of either the gradient ordering or gradientattention-product ordering in many cases. In addition, although the product-based ranking often (but not always) requires slightly fewer removed items than the gradient ranking, we see that the purely gradient-based ranking ignoring attention magnitudes comes quite close to it, far outperforming purely attention-based orderings. For ten of our 16 models with encoders, removing by gradient found a smaller decision-flipping set of items than attention for over 50% of instances in that model's test set, with that percentage often being much higher. In fact, for every model with an encoder that we tested, there were at least 1.6 times as many test instances where the purely gradientbased ranking managed a decision flip faster than the attention-based ranking than vice versa.
Is Attention Interpretable?
We do not claim that ranking importance by either descending gradients or descending gradientattention products is optimal, but in many cases they discover much smaller decision-flipping sets of items than attention weights. Therefore, ranking representations in descending order by attention weight clearly fails to uncover a minimal set of decision-flipping information much of the time, which is a warning sign that we should be skeptical of trusting groups of attention weight magnitudes as importance indicators.
Is Attention Interpretable?
For all ordering schemes we tried, we were struck by the large fraction of items that had to be removed to achieve a decision flip in many models. This is slightly less surprising for the HANs, as they compute attention over shorter sequences of sentences (see Table 1). For the FLAN models, though, this result is highly unexpected. The sequences across which FLANs compute attention are usually hundreds of tokens in length, meaning most attention weights will likely be minuscule.
Is Attention Interpretable?
The distributions of tokens removed by our dif- Figure 5: The distribution of fractions of items removed before first decision flips on three model architectures under different ranking schemes. Boxplot whiskers represent the highest/lowest data point within 1.5 IQR of the higher/lower quartile, and dataset names at the bottom apply to their whole column. In several of the plots, the median or lower quartile aren't visible; in these cases, the median/lower quartile is either 1 or very close to 1. ferent orderings that we see for the FLANrnns in Figure 5 are therefore remarkably high, especially given that all of our classification tasks have at least five output classes. We also note that due to the exponential nature of the softmax, softmax attention distributions typically contain only a few high-weighted items before the calculated weights become quite small, which can be misleading. In many cases, flipping the model's original decision requires digging deep into the small attention weights, with the high-weighted components not actually being the reason for the decision.
Is Attention Interpretable?
For several of our models, especially the FLANs (which typically compute attention over hundreds of tokens), this fact is concerning from an explainability perspective. Lipton (2016) describes a model as "transparent" if "a person can contemplate the entire model at once." Applying this insight to the explanations suggested by attention, if an explanation rests on simultaneously considering hundreds of attended tokens necessary for a decision-even if that set were minimal-that would still raise serious transparency concerns.
Is Attention Interpretable?
One last question we consider is whether the large number of items that are removed before decision flips can be explained in part by the scope of each model's contextualization. In machine translation, prior work has observed that recurrent encoders over a full sequence can "shift" tokens' signal in ways that cause subsequent attention layers to compute unintuitive off-by-one alignments (Koehn and Knowles, 2017). We hypothesize that in our text classification setting, the bidirectional recurrent structure of the HANrnn and FLANrnn encoders might instead be redistributing operative signal from a few informative input tokens across many others' contextualized representations.
Is Attention Interpretable?
Comparing the decision flip results for the FLANconvs in Figure 5 to those for the FLANrnns supports this theory. We notice decision flips happening much faster than for either of the RNN-based model architectures, indicating that the biRNN effectively does learn to widely redis- Figure 6: The distribution of fractions of items removed before decision flips on the encoderless model architectures under different ranking schemes. The Amazon FLANnoenc results have a very long tail; using the legend's order of rankings, the percentage of test instances with a removed fraction above 0.50 for that model is 12.4%, 2.8%, 0.9%, and 0.5%, respectively. tribute the classification signal. In contrast, the convolutional encoders only allow contextualization with respect to an input token's two neighbors to either side. We see similar results when comparing the two HAN architectures, albeit much more weakly (see Figure 10 in Appendix A.2); this is likely due to the smaller number of tokens being contextualized by the HANs (sentence representations instead of words), so that contextualization with respect to a token's close neighbors encompasses a much larger fraction of the full sequence. We see this difference even more strongly when we compare to the encoderless model architectures, as shown in Figure 6. Compared to both other model architectures, we see the fraction of necessary items to erase for flipping the decision plummet. We also see random orderings mostly do better than before, indicating more brittle decision boundaries, especially on the Amazon dataset. 6 In this situation, we see attention magnitudes generally indicate importance on par with (or better than) gradients, but that the product-based order-6 This is likely due to the fact that with no contextualization, the final attended representations are just a linear combination of the input embeddings, so the embeddings themselves are responsible for learning to directly encode a decision. Since Amazon has the largest ratio of documents (which probably vary in their label) to unique word embeddings by a factor of more than two times any other dataset's, and the final attended representations in the FLANnoencs are unaggregated word embeddings, it stands to reason that the lack of encoders would be a much bigger obstacle in its case.
Is Attention Interpretable?
ing is still often a more efficient explanation.
Is Attention Interpretable?
While these differences themselves are not an argument against attention's interpretability, they highlight the distinction between attention's weighting of intermediate, contextualized representations and the model's use of the original input tokens themselves. Our RNN-based models' ability to maintain their original decision well past the point at which models using only local or no context have lost the signal driving their original decisions confirms that attention weights for a contextualized representation do not necessarily map neatly back to the original tokens. This might at least partly explain the striking near-indifference of the model's decision to the contributions of particular contextualized representations in both our RNN-based models and in Jain and Wallace (2019), who also use recurrent encoders.
Is Attention Interpretable?
However, the results from almost all models continue to support that ranking importance by attention is still not optimal; our non-random alternative rankings still uncover many cases where fewer items could be removed to achieve a decision flip than the attention weights imply.
Is Attention Interpretable?
There are important limitations to the work we describe here, perhaps the most important of which is our focus on text classification. By choosing to fo-cus on this task, we use the fact that decision flips are often not trivially achieved to ground our judgments of importance in model decision changes. However, for a task with a much larger output space (such as language modeling or machine translation) where almost anything might flip the decision, decision flips are likely too coarse a signal to identify meaningful differences. Determining an analogous informative threshold in changes to model outputs would be key to expanding this sort of analysis to other groups of models.
Is Attention Interpretable?
A related limitation is our reliance in many of these tests on a fairly strict definition of importance tied to the output's argmax; an alternative definition of importance might assert that the highest attention weights should identify the most influential representations in pushing towards any output class, not just the argmax. Two of the core challenges that would need to be solved to test for how well attention meets this relaxed criterion would be meaningfully evaluating a single attended item's "importance" to multiple output classes for comparison to other attended items and, once again, determining what would truly indicate being "most influential" in the absence of decision flips as a guide to the output space.
Is Attention Interpretable?
Also, while we explore several model architectures in this work, there exist other attention functions such as multi-headed and scaled dot-product (Vaswani et al., 2017), as well as cases where a single attention layer is responsible for producing more than one attended representation, such as in self-attention (Cheng et al., 2016). These variants could have different interpretability properties. Likewise, we only evaluate on final layers of attention here; in large models, lower-level layers of attention might learn to work differently.
Is Attention Interpretable?
We have adopted an erasure-based approach to probing the interpretability of computed attention weights, but there are many other possible approaches. For example, recent work has focused on which training instances (Koh and Liang, 2017) or which human-interpretable features were most relevant for a particular decision (Ribeiro et al., 2016;Arras et al., 2016). Others have explored alternative ways of comparing the behavior of proposed explanation methods (Adebayo et al., 2018). Yet another line of work focuses on aligning models with human feedback for what is interpretable (Fyshe et al., 2015;Subramanian et al., 2017), which could refine our idea of what defines a highquality explanation derived from attention.
Is Attention Interpretable?
Finally, another direction for future work would be to extend the importance-ranking comparisons that we deploy here for evaluation purposes into a method for deriving better, more informative rankings, which in turn could be useful for the development of new, more interpretable models.
Is Attention Interpretable?
It is frequently assumed that attention is a tool for interpreting a model, but we find that attention does not necessarily correspond to importance. In some ways, the two correlate: comparing the highest attention weight to a lower weight, the high attention weight's impact on the model is often larger. However, the picture becomes bleaker when we consider the many cases where the highest attention weights fail to have a large impact. Examining these cases through multi-weight tests, we see that attention weights often fail to identify the sets of representations most important to the model's final decision. Even in cases when an attention-based importance ranking flips the model's decision faster than an alternative ranking, the number of zeroed attended items is often too large to be helpful as an explanation. We also see a marked effect of the contextualization scope preceding the attention layer on the number of attended items affecting the model's decision; while attention magnitudes do seem more helpful in uncontextualized cases, their lagging performance in retrieving decision rationales elsewhere is cause for concern. What is clear is that in the settings we have examined, attention is not an optimal method of identifying which attended elements are responsible for an output. Attention may yet be interpretable in other ways, but as an importance ranking, it fails to explain model decisions. 
Is Attention Interpretable?
We lowercased all tokens during preprocessing and used all hyperparameters specified in (Yang et al., 2016), except for those related to the optimization algorithm or, in the case of the convolutional or no-encoder models, the encoder. For each convolutional encoder, we trained two convolutions: one sweeping over five tokens, and one sweeping over three. As the output representation of token x, we then concatenated the outputs of the five-token and three-token convolutions centered on x. Unless otherwise noted, to train each model, we used Adam (Kingma and Ba, 2014) with gradient clipping of 10.0 and a patience value of 5, so we would stop training a model if five epochs elapsed without any improvement in validation set accuracy. In addition, for each model, we specified a learning rate for training, and dropout before each encoder layer (or attention layer, for the encoderless models) and also within the classification layer. For the HAN models, these are the values we used:
Is Attention Interpretable?
•  Trained model classification accuracies are reported in Table 3. We note that our IMDB data and Yelp data are different sets of reviews from those used by Yang et al. (2016), so our reported performances are not directly comparable to theirs.
Is Attention Interpretable?
We were unable to reach a comparable performance for the Amazon dataset (and Yelp dataset, although different) to that in (Yang et al., 2016). We suspect that this is due to not pretraining the word2vec embeddings used by the model for long enough, combined with memory limitations on our hardware that necessitated decreasing our batch size in many cases. However, as noted in section 3, the analysis that we perform does not depend on model accuracy. It's also worth noting that for the datasets for which we are able to get results that either pass or come close to the accuracies listed in the original HAN paper, the patterns we see in the results for the tests that we run are the same as the patterns that we see for the others.
Is Attention Interpretable?
Here we include the full sets of result plots for all models for all tests we describe in the paper, in order of appearance.
Is Attention Interpretable?
In Figure 7, we see that the majority of ∆JS values continue to fall above 0, and that most are still close to 0. One point not stated in the paper, though, is that the upswing in ∆JS values as the difference between i * 's weight and a randomly chosen weight increases tends to occur slightly earlier for models with less contextualization, implying that the improving efficiency of the attention-based ranking at flipping the decision as contextualization scope shrinks is also reflected in single-weight test results.
Is Attention Interpretable?
Looking at where negative ∆JS values tend to occur in Figure 8, we once again see that they tend to cluster around cases where the difference between the highest and randomly chosen attention weights is close to 0. There are some exceptions, however; perhaps the most obvious are the fat tails of these counts for the Yahoo Answer HAN models. Considering the highest-attentionweight ranking of importance for all Yahoo Answers HAN models in Figure 10 struggle in flipping the decision quickly, it may be that attention is less helpful than usual in identifying importance in its case, which could explain this discrepancy.
Is Attention Interpretable?
In Figure 9, we list contingency tables for all i *versus-random single-weight decision-flip tests. We continue to see higher values overall in our blue cells than orange, as described in section 4.2. The most general change we notice across all the tables is that in the encoderless case, there are more test instances (often many more) where at least one of i * or our random attended item flipped the decision than for any other architecture, except in the case of the Yahoo Answers FLAN. Thinking about why this might be, we recall that in the encoderless case, word embeddings are much more directly responsible for encoding a decision. Yahoo Answers is our only topic classification dataset, where keywords like "computer" or "basketball" might be much clearer indicators of a topic than, say, "like" or "love" would be indicators of a rating of 8 versus 9. This likely leads to much less certain decisions being encoded in the word embeddings of the non-Yahoo Answers datasets. For all other models, and in the case where potentially contradictory Yahoo Answers word embeddings are blended together before the final layer of attention (its HANnoenc), it is likely that decisions are simply more brittle overall.
Is Attention Interpretable?
Finally, in Figure 10, we include the full set of fraction-removed distributions for the first decision flips reached under the different rankings we explored.
Is Attention Interpretable?
Besides the tests we describe in the main paper, some of the other tests that we ran provide additional insights into our results. We briefly describe those here.
Is Attention Interpretable?
In Figure 11, we provide the distributions of the original attention probability distributions that were zeroed at the point when different ranking schemes achieved their first decision flips. (Equivalently, these are the distributions of the sums of the zeroed attention weights described in Figure  10, only without repeated normalization.) We include these results to give a sense of which attention magnitudes the different rankings typically place towards the top. We notice that this probability mass required to change a decision is often quite high, which is unsurprising for the attentionbased ranking, given that it frequently requires removing many items to flip decisions and attention distributions tend to have just a few high weights.
Is Attention Interpretable?
Besides that, the main takeaway that we see here is that for most models, the distribution of attention probability masses zeroed by our gradientbased ranking or our product-based ranking is often shifted down by around 0.25 or more compared to the corresponding attention probability mass distribution for the attention-based ranking, which is a fairly large difference. This would seem to imply that these alternative rankings (which usually flip decisions faster) tend to differ in relatively substantial ways from the rankings suggested by the pure attention weights, not just in the long tail of their orderings, which is another warning sign against attention's interpretability.
Is Attention Interpretable?
The final set of tests that we include in Figures 12 and 13 consist of rerunning our singleweight decision-flip tests on the single "most important" attention weights in their respective attention distributions as suggested by our alternative rankings (gradient-based and product-based rankings) instead of attention magnitudes. These results serve two functions: first, they imply still more information about when the top weight suggested by an alternative, faster-decision-flipping ranking differs from the top attention weight. Intuitively, if we observe large differences between the sum of the "yes" row for one contingency table and the "yes" rows for the other rankings' tables on that same model, this is likely due to differences in the frequencies with which the highestranked items achieve a decision flip, indicating differences in highest-ranked items ("likely" because of the noise added by the random sampling).
Is Attention Interpretable?
The second piece of information that these tests provide is a lower bound (via the sum of the "yes" rows) for the number of cases where rankings flip a decision as quickly as possible (i.e., in the first item). For context, the sum of the "yes" row is higher than the corresponding sum in Figure 9 for all contingency tables using our product-based ordering. For the gradient-based ordering, however, this sum is actually lower than for the attentionbased ranking's tables in 14 out of our 24 models. This tells us that our gradient-based method often finds fewer single-item ways of flipping decisions than the attention-based ranking, so in order to achieve its more efficient overall distribution of flips that we see for many models in Figure 10, it must usually flip decisions faster than attention in cases where both its ranking and the attentionbased ranking require multiple removed weights.  
On the Sensitivity and Stability of Model Interpretations in NLP
As complex NLP models are widely deployed in real-world applications, there is an increasing interest in understanding how these models come to certain decisions. As a result, the line of research on interpretation techniques grows rapidly, facilitating a broad range of model analysis, from building user trust on models (Ribeiro et al., 2016;Hase and Bansal, 2020) to exposing subtle biases (Zhao et al., 2017;Doshi-Velez and Kim, 2017).
On the Sensitivity and Stability of Model Interpretations in NLP
In this paper, we focus on post-hoc interpretations in NLP. Given a trained model and a specific input text, post-hoc interpretations assign an importance score to each token in the input which indicates its contribution to the model output. Cur-rent methods in this direction can be roughly divided into three categories: gradient-based methods (Simonyan et al., 2014;Li et al., 2016); referencebased methods (Sundararajan et al., 2017;Shrikumar et al., 2017); and perturbation-based methods (Zeiler and Fergus, 2014;Ribeiro et al., 2016).
On the Sensitivity and Stability of Model Interpretations in NLP
Despite the emergence of new techniques, one critical issue is that there is little consensus on how to define and evaluate the faithfulness of these techniques, i.e., whether they reflect the true reasoning process by a model. A widely employed criterion, especially in NLP, is the removal-based criterion (DeYoung et al., 2020), which removes or only preserves a set of tokens given by interpretations and measures how much the model prediction would change. However, as pointed out in prior work (Bastings and Filippova, 2020;Ancona et al., 2018), the corrupted version of an input produced during evaluations falls out of the distribution that models are trained on, and thus results in an inaccurate measurement of faithfulness. This limitation prevents removal-based metrics from being used as the golden standard for evaluating interpretations. To remedy this, we complement the removal-based criterion with two other criteria, sensitivity and stability, which are overlooked in prior works.
On the Sensitivity and Stability of Model Interpretations in NLP
Sensitivity is based on the notion that models should be more sensitive to perturbations on tokens identified by a faithful explanation. In contrast to the removal-based criterion, which completely removes important tokens, the sensitivity criterion adds small but adversarial perturbations in a local region of the token embedding, and thus preserves the structure of input sentences as well as interactions between context words. This criterion is recently discussed in  in computer vision, while we provide comprehensive analyses on various NLP models and tasks. Note that while the removal-based criterion asks the question: if some important tokens did not 'exist', what would happen, the sensitivity criterion asks: if some im-portant tokens were 'changed' adversarially, what would happen.
On the Sensitivity and Stability of Model Interpretations in NLP
Stability assumes that a faithful interpretation should not produce substantially different explanations for two inputs that the model finds similar. There are several attempts to generate such a pair of inputs. The most relevant one is Ghorbani et al. (2019). However, their method is only applicable to differentiable interpretations. Our work proposes a new paradigm based on adversarial word substitution that employs a black-box algorithm to generate a semantically related neighbor of the original input, which is specially designed for NLP and applicable to all interpretations techniques.
On the Sensitivity and Stability of Model Interpretations in NLP
The above two metrics highlight the connection between interpretability and robustness. Experiments show that interpretations which perform well on the removal-based criterion might not do well on the new criteria. Motivated by the limitations of existing interpretations and the desiderata of sensitivity, we propose robustness-based methods, based on projected gradient descent (PGD) attacks (Madry et al., 2018) and certifying robustness (Jia et al., 2019;Huang et al., 2019;Xu et al., 2020). We demonstrate that the new methods achieve top performance under sensitivity and stability. Moreover, as a simple improvement to gradient-based methods, our methods avoid the gradient saturation issues of gradient-based methods under the removal-based criterion.
On the Sensitivity and Stability of Model Interpretations in NLP
Another limitation of removal-based metrics emerges when interpreting dependency parsingwhen input tokens are removed, the tree structure is drastically changed and a model might not be able to produce a meaningful parse tree. Thus, there are little discussion for dependency parsing interpretations. In this paper, we propose a new paradigm to interpret dependency parsers leveraging prepositional phrase (PP) attachment ambiguity examples. To our best knowledge, this is the first work to study interpretations on dependency parsing. We demonstrate that sensitivity does not change the output tree structure as much as removal-based ones do, and provide analyses for interpretation methods with our paradigm and metrics.
On the Sensitivity and Stability of Model Interpretations in NLP
Our contributions can be summarized as follows. 1. We discuss two overlooked notions of faithfulness in NLP interpretations. Our notions emphasize the connection between interpretability and robustness. We systematically evaluate interpretations under these notions, including existed removalbased ones. The code for this paper could be found at https://github.com/uclanlp/ NLP-Interpretation-Faithfulness.
On the Sensitivity and Stability of Model Interpretations in NLP
2. We propose new robustness-based interpretations inspired by the sensitivity metric and demonstrate their effectiveness under both sensitivity and stability. 3. We propose a novel paradigm to evaluate interpretations on the dependency parsing task.
On the Sensitivity and Stability of Model Interpretations in NLP
A faithful post-hoc interpretation identifies the important parts of the input a model prediction relies on. Let x = [x 1 ; x 2 ; . . . ;
On the Sensitivity and Stability of Model Interpretations in NLP
x n ] be a sequence of tokens. e (·) denotes the token embedding function. An NLP model f takes the embedding matrix e (x) ∈ R n×d as input and provides its prediction f (e (x)) = y. Let s y (e (x)) denote the output score of f (e (x)) on y. The exact form of s y (e (x)) is defined in Appendix D. An interpretation assigns an importance score to each token which indicates its contribution to the model decision.
On the Sensitivity and Stability of Model Interpretations in NLP
We first review the well-established removalbased criterion and emphasize its relation to the two criteria defined in this paper 1) sensitivity, and 2) stability, for which we propose novel paradigms to adapt them to various NLP tasks. Removal-based Criterion A well-established notion of interpretation faithfulness is that the presence of important tokens should have more meaningful influence on the model's decision than random tokens, quantified by the removal-based criterion. We adopt the comprehensiveness and the sufficiency score in DeYoung et al. (2020). The comprehensiveness score measures how much the model performance would drop after the set of "relevant" tokens identified by an interpretation is removed. A higher comprehensiveness score suggests the tokens are more influential to the model output, and thus a more faithful explanation. The sufficiency score measures to what extent the original model performance is maintained when we solely preserve relevant tokens. A lower sufficiency score means less change in the model prediction, and thus a more faithful explanation. See DeYoung et al. (2020) for detailed definitions. Note that completely removing input tokens produces incomplete texts. Large perturbation of this kind lead to several issues as pointed out by prior studies (Feng et al., 2018;Bastings and Filippova, 2020).
On the Sensitivity and Stability of Model Interpretations in NLP
Ours: Sensitivity Instead of removing important tokens, the sensitivity criterion adds local but adversarial noise to embedding vectors of the important tokens and measures the magnitude of the noise needed to change the model prediction. This is inspired by the notion that models should be more sensitive to perturbations being added to relevant tokens compared to random or irrelevant tokens. From the adversarial robustness perspective , this notion implies that by perturbing the most relevant tokens, we can reach the local decision boundary of a model with the minimum perturbation magnitude.
On the Sensitivity and Stability of Model Interpretations in NLP
Given the sequence of relevant tokens r k , sensitivity adds perturbation to its embedding e (r k ) but keeps the remaining token embeddings unchanged. Then, it measures the minimal perturbation norm, denoted as ϵ r k , that changes the model prediction for this instance:
On the Sensitivity and Stability of Model Interpretations in NLP
where ∥·∥ F is the Frobenius norm of a matrix, and δ r k ∈ R n×d denotes the perturbation matrix where only the columns for tokens in r k have non-zero elements. Since the exact computation of ϵ r k is intractable, we use the PGD attack (Madry et al., 2018) with a binary search to approximate ϵ r k . A lower ϵ r k suggests a more faithful interpretation. In practice, we vary the size of r k , compute multiple ϵ r k , and summarize them with the area under the curve (AUC) score.
On the Sensitivity and Stability of Model Interpretations in NLP
Ours: Stability Another desired property of faithfulness is that a faithful interpretation should not give substantially different importance orders for two input points that the model finds similar. To construct a pair of similar inputs, we propose to generate contrast examples to the original one by synonym substitutions. A contrast example of x,x, satisfies (1) has at most k different but synonymous tokens with x; (2) the prediction score atx changes less than τ compared to the score at x. The goal of these two conditions is to generate (almost) natural examples where the changes of model outputs are smaller than a threshold τ . Given all contrast examples, we search for the one that leads to the largest rank difference D between the importance order for x, m (x) and the alternated order m (x):
On the Sensitivity and Stability of Model Interpretations in NLP
s.t. |s y (e (x)) − s y (e (x))| ≤ τ, ∥x −x∥ 0 ≤ k.
On the Sensitivity and Stability of Model Interpretations in NLP
Specifically, we first extract synonyms for each token x i following Alzantot et al. (2018). Then, in the decreasing order of m (x), we greedily search for a substitution of each token that induces the largest change in m (x) and repeat this process until the model output score changes by more than τ or the pre-defined constraint k is reached. Finally, we measure the difference D between two importance ranks using Spearman's rank order correlation (Spearman, 1961). We call this criterion stability. A higher score indicates that the ranks between this input pair are more similar, and thus a more faithful interpretation.
On the Sensitivity and Stability of Model Interpretations in NLP
Note that instead of using the gradient information of interpretation methods to perturb importance ranks like Ghorbani et al. (2019), our algorithm treats interpretations as black-boxes, which makes it applicable to non-differentiable ones. Also, compared to Ding and Koehn (2021), who manually construct similar input pairs, our method is a fully automatic one as suggested by their paper.
On the Sensitivity and Stability of Model Interpretations in NLP
Experiments indicate that existing methods do not work well with the sensitivity and stability metrics (Sec. 4.2). In this section, we define a new class of interpretation methods by adopting techniques in adversarial robustness to remedy this. We first give a brief review of existing interpretation approaches and then introduce our new methods.
On the Sensitivity and Stability of Model Interpretations in NLP
We roughly divide the existing methods into three categories: gradient-based methods, referencebased methods, and perturbation-based methods, and discuss the representatives of them. Gradient-based methods The first class of methods leverage the gradient at each input token. To aggregate the gradient vector at each token into a single importance score, we consider two methods: 1) using the L 2 norm, ∂sy(e(x)) ∂e(x i ) 2
On the Sensitivity and Stability of Model Interpretations in NLP
, referred to as Vanilla Gradient (VaGrad) (Simonyan et al., 2014), and 2) using the dot product of gradient and input, ∂sy(e(x)) ∂e(x i ) ⊤ ·e (x i ), referred to as Gradient · Input (GradInp) (Li et al., 2016). Reference-based methods These methods distribute the difference between model outputs on a reference point and on the input as the importance score for each token. We consider Inte-grated Gradient (IngGrad) (Sundararajan et al., 2017) and DeepLIFT (Shrikumar et al., 2017). Ing-Grad computes the linear intergral of the gradients from the reference point to the input. DeepLIFT decomposes the difference between each neuron activation and its 'reference activation' and back-propagates it to each input token. We use DeepLIFT with the Rescale rule. Note DeepLIFT diverges from IngGrad when multiplicative interactions among tokens exist (Ancona et al., 2018). Perturbation-based methods Methods in this class query model outputs on perturbed inputs. We choose Occlusion (Zeiler and Fergus, 2014) and LIME (Ribeiro et al., 2016). Occlusion replaces one token at a time by a reference value and uses the corresponding drop on model performance to represent the importance of each token. LIME uses a linear model to fit model outputs on the neighborhood of input x and represents token importance by the weights in the trained linear model.
On the Sensitivity and Stability of Model Interpretations in NLP
We propose two methods inspired from the PGD attack (Madry et al., 2018) and the certifying robustness algorithms (Xu et al., 2020) in adversarial robustness. VaPGD and PGDInp The PGD attack in adversarial robustness considers a small vicinity of the input and takes several "mini-steps" within the vicinity to search for an adversarial example. Consider the token embeddings for the input x, we perform t iterations of the standard PGD procedure starting from e (0) = e (x): e (j) = P e (j−1) −α∇s y e (j−1) , j = 1, 2, . . . , t.
On the Sensitivity and Stability of Model Interpretations in NLP
P represents the operation that projects the new instance at each step back to the vicinity of e (x), and α is the step size.
On the Sensitivity and Stability of Model Interpretations in NLP
Intuitively, e (t) − e (x) tells us the descent direction of model confidence. Similar to the gradientbased methods, the importance of each token x i can be either represented by e (t)
On the Sensitivity and Stability of Model Interpretations in NLP
i is the i-th column in e (t) , referred to as Vanilla PGD (VaPGD), or by e (x i ) − e (t) i ⊤ · e (x i ), referred to as PGD · Input (PGDInp)
On the Sensitivity and Stability of Model Interpretations in NLP
Note that different from the PGD attack we use for approximating the sensitivity criterion, we manually decide the magnitude of the vicinity of e (x) instead of using a binary search. We add perturbations to the whole sentence at the same time. Also, the final e (t) does not necessarily change the model prediction. See Appendix B for details.
On the Sensitivity and Stability of Model Interpretations in NLP
Certify Certifying robustness algorithms also consider a vicinity of the original input and aim to provide guaranteed lower and upper bounds of a model output within that region. We use the linear relaxation based perturbation analysis (LiRPA) discussed in Xu et al., 2020). LiRPA looks for two linear functions that bound the model. Specifically, LiRPA computes W , W , b, and b that satisfy
On the Sensitivity and Stability of Model Interpretations in NLP
for any point e (x ′ ) that lies within the L 2 ball of e (x) with size δ. We use the IBP+backward method in Xu et al. (2020). It uses Interval Bound Propagation (Gowal et al., 2018;Mirman et al., 2018) to compute bounds of internal neurons of the model and then constructs the two linear functions with a bound back-propagation process (Zhang et al., 2018;Singh et al., 2019). Finally, the importance score of the i-th token in the input is represented by W i · e (x i ), where W i is the i-th row of W . We call this method Certify.
On the Sensitivity and Stability of Model Interpretations in NLP
Robustness-based vs. Gradient-based Gradientbased methods provide a linear approximation of the model decision boundary at the single input, which is not accurate for non-linear models. Robustness-based methods instead search multiple steps in neighbors and approximate the steepest descent direction better. We also empirically show that robustness-based methods avoid the saturation issue of gradient-based methods, i.e, gradient becomes zero at some inputs. See Appendix H. Note that VaPGD (PGDInp) degrades to VaGrad (Grad-Inp) when the number of iterations is 1.
On the Sensitivity and Stability of Model Interpretations in NLP
Robustness-based vs. IngGrad IngGrad leverages the average gradient in a segment between the input and a reference. It is likely to neglect local properties desired by the sensitivity criterion. Robustness-based methods instead search in the vicinity of the input, and thus local properties are better preserved. See results in Sec. 4.2.
On the Sensitivity and Stability of Model Interpretations in NLP
In this section, we present the results on text classification tasks under the three criteria. We find that the correlation between interpretation faithfulness based on different criteria are relatively low in some cases. Results verify the effectiveness of our new methods.
On the Sensitivity and Stability of Model Interpretations in NLP
Datasets We conduct experiments on three text classification datasets: SST-2 (Socher et al., 2013), Yelp (Zhang et al., 2015), and AGNews (Zhang et al., 2015) following Jain and Wallace (2019)'s preprocessing approach. All of them are converted to binary classification tasks. SST-2 and Yelp are sentiment classification tasks where models predict whether a review is negative (0) or positive (1). AGNews is to discriminate between world (0) and business (1) articles. See Appendix A for statistics of the three datasets. When evaluating interpretation methods, for each dataset, we select 200 random samples (100 samples from class 0 and 100 samples from class 1) from the test set. Models For text classification, we consider two model architectures: BERT (Devlin et al., 2019) and BiLSTM (Hochreiter and Schmidhuber, 1997). Interpretation Methods Besides our robustnessbased interpretations PGDInp, VaPGD, and Certify, we experiment with six others from three existing categories: VaGrad, GradInp (gradientbased); IngGrad, DeepLIFT (reference-based); and Occlusion, LIME (perturbation-based). We also include a random baseline Random that randomly assigns importance scores. We use comprehensiveness (Comp.), sufficiency (Suff.), sensitivity (Sens.), and stability (Stab.) as metrics.
On the Sensitivity and Stability of Model Interpretations in NLP
See Appendix A∼C for experimental details.
On the Sensitivity and Stability of Model Interpretations in NLP
Overall Results Results of interpretations for BERT and BiLSTM are presented in Table 1 and 2. The interpretations' performance are averaged over three runs on models trained from different random seeds. Results verify the effectiveness of our proposed robustness-based methods. Specifically, VaPGD achieves the best performance under the sensitivity and the stability criteria for both BERT and BiLSTM. Our methods also outperform their gradient-based counterparts under removal-based criteria. Especially, when interpreting BERT on SST-2 and AGNews, GradInp has near random performance. PGDInp can avoid these unreasonable behaviors. See Appendix H for a qualitative study on this, where we find PGDInp does not suffer from the saturation issue as GradInp. Also notice that the superior performance of robsutness-based methods are consistent on BERT and BiLSTM+GloVe, which demonstrate that it is not influenced by the embeddings being used.
On the Sensitivity and Stability of Model Interpretations in NLP
The film's center will not hold .
On the Sensitivity and Stability of Model Interpretations in NLP
The film's center will not hold .
On the Sensitivity and Stability of Model Interpretations in NLP
Comp. = 0.377 Sens. = 0.252
On the Sensitivity and Stability of Model Interpretations in NLP
The film's center will not hold . However, the performance of other methods tend to be inconsistent under different measurements. For example, under the removal-based criterion, IngGrad performs well for BiLSTM, which gives four out of six best numbers. But, IngGrad has very limited performance under the sensitivity metric, especially for BiLSTM on SST-2 and Yelp. Similar issues exist for LIME and Occlusion. Also, one might fail to recognize the faithfulness of VaPGD by solely looking at the removal-based criterion. Thus, when deploying interpretation methods on real tasks, we advocate for a careful selection of the method you use based on the underlying faithfulness notion that aligned with your goal.
On the Sensitivity and Stability of Model Interpretations in NLP
Performance Curves To show how the size of the relevant set affects interpretation performance, we plot the comprehensiveness and the sensitivity curves when increasing the number of tokens being removed (perturbed  set is increased, the performance of IngGrad under the comprehensiveness metric and the performance of VaPGD under the sensitivity metric gradually surpass Occlusion and other methods. This implies that the two methods are better at identifying a relevant set with more tokens.
On the Sensitivity and Stability of Model Interpretations in NLP
Interpolation Analysis To check whether the comprehensiveness and sensitivity scores can reflect the relative importance of each token in the relevant set, we conduct an interpolation analysis that gradually replaces each token in the relevant set with a random token outside of the set. Specifically, we select 50 examples from SST-2 and test on BERT with relevant sets given by LIME and VaPGD. For each example, we extract a relevant set consists of the top four most important tokens and gradually replace each token, from the least to the most important one, with a random token. We denote the relevant set at each step as S 0 , S 1 , ..., S 4 , where S 0 is the original relevant set containing the top four tokens and S 4 is the set of four random tokens. The performance change
On the Sensitivity and Stability of Model Interpretations in NLP
where M is the comprehensiveness or sensitivity score. We expect that a good metric should induce a monotonically increasing function f . Further, f should be strictly convex as that indicates the importance of each token is different.
On the Sensitivity and Stability of Model Interpretations in NLP
We plot the curve in Figure 3. Results show that both the comprehensiveness and sensitivity metrics generate a monotonically increasing function, which indicates that they fully consider each token in the relevant set. Also, notice that based on the comprehensiveness metric, the contribution of each token tends to distribute evenly within the relevant set, which contradicts the fact that tokens in the set have different contribution to the prediction, while the importance rank is better preserved based on the sensitivity metric,.
On the Sensitivity and Stability of Model Interpretations in NLP
Different Notions of Faithfulness Finally, we qualitatively study the notions of faithfulness defined by comprehensiveness (comp.) and sensitivity (sens.), and discuss two main differences.
On the Sensitivity and Stability of Model Interpretations in NLP
First, comp. removes important tokens during  Figure 3: Interpolation between the relevant set and a random set. The relevant set for Comprehensiveness is given by LIME, and the set for Sensitivity is given by VaPGD. We find that Sensitivity better preserves the relative importance for each token in the relevant set. evaluations, which could possibly break the interaction between removed tokens and context tokens, and underestimate the importance of context tokens. In Figure 1(a), the tokens 'not' and 'hold' together determine the negative sentiment of the sentence. Sens. considers both 'not' and 'hold' as important tokens as one expects. However, comp. regards 'hold' less important than 'will'.
On the Sensitivity and Stability of Model Interpretations in NLP
Second, sens. measures token importance by how much model performance would change after 'adversarially perturbing' that token. In this sense, both positive and negative pertinent tokens will be deemed important. In contrast, comp. only considers positive pertinent ones. In Figure 1(b), which is predicted as positive, removing the negative verb 'hate' would not influence model performance much. However, adversarially perturbing 'hate' (e.g. change 'hate' to a more negative verb) might change the model prediction from positive to negative. Thus, sens. prefers interpretations that identify 'hate' as an important token like VaPGD.
On the Sensitivity and Stability of Model Interpretations in NLP
The full version of Figure 1
On the Sensitivity and Stability of Model Interpretations in NLP
Some contrast examples generated for the stability criterion are presented in Appendix F.
On the Sensitivity and Stability of Model Interpretations in NLP
Structured prediction tasks are in the center of NLP applications. However, applying interpretation methods and criteria to these tasks are difficult because 1) the required output is a structure instead of a single score. It is hard to define the contribution of each token to a structured output, and 2) compared to text classification tasks, removing parts of the input like what removal-based criteria do, would cause more drastic changes to model predictions as well as the groundtruth. Therefore, existing works often conduct experiments only on binary or multi-class text classification tasks. To remedy these issues, we investigate interpretations for dependency parsing, with an special focus on analyzing how models resolve the PP attachment ambiguity, which avoids interpreting the structured output as a whole. We show that our sensitivity metric is a better metric for dependency parsing as it causes negligible changes to model outputs compared to removal-based metrics.
On the Sensitivity and Stability of Model Interpretations in NLP
Our paradigm focuses on the PP attachment ambiguity, which involves both syntactic and semantics considerations. A dependency parser needs to determine either the preposition in PP attaches to the preceding noun phrase NP (NP-attachment) or the verb phrase VP (VP-attachment) (Hindle and Rooth, 1993). The basic structure of ambiguity is VP -NP -PP. For example, in the sentence I saw a cat with a telescope, a parser uses the semantics of the noun phrase a telescope to predict the head of with, which is saw. If we change a telescope to  a tail, the head of with would become the preceding noun cat. We will later call nouns in PPs like telescope "disambiguating nouns", as they provide semantic information for a parser to disambiguate PP attachment ambiguity. The main advantage of this paradigm is that disambiguating nouns can be viewed as "proxy groundtruths" for faithfulness as parsers must rely on them to make decisions.
On the Sensitivity and Stability of Model Interpretations in NLP
We use DeepBiaffine, a graph-based dependency parser as the target model (Dozat and Manning, 2017). We extract 100 examples that contain the PP attachment ambiguity from the English Penn Treebank converted to Stanford Dependencies 3.5.0 (PTB-SD). We consider the same interpretation methods as before, and they assign an importance score to each token in the sentence to indicate how much it impacts the model prediction on PP attachment arcs. We test the faithfulness of the attributions using comprehensiveness and sensitivity. See Appendix A∼C for details.
On the Sensitivity and Stability of Model Interpretations in NLP
Results are shown in Table 3. Similar to the results on text classification tasks, we find that perturbation-based methods like LIME and Occlusion perform well under the comprehensiveness score, while VaPGD performs the best under sensitivity. PGDInp and Certify are slightly better than GradInp under both the two metrics. Qualitatively, we find that according to interpretation methods, important tokens for a PPattachment decision converge tothe preposition itself, the preceding noun or verb, and the disambiguating noun. This is close to human expectations. An example is shown in Appendix E.  Table 4: Similarity between the parser outputs before and after applying the evaluation metric. We show that sensitivity changes the global model output less.
On the Sensitivity and Stability of Model Interpretations in NLP
Metric Check Removing even a small piece of inputs breaks the dependency tree. It will be hard to distinguish either the decision process behind the model has changed or the removal of important tokens actually causes the performance drop. Thus, we expect a better metric to have less influence on the tree structure of a sentence. In Table 4, we show that evaluating interpretations with sensitivity leads to smaller changes in the output dependency tree compared to comprehensiveness, suggesting sensitivity a more compatible metric for the dependency parsing task interpretations.
On the Sensitivity and Stability of Model Interpretations in NLP
Disambiguating Noun Analysis Disambiguating nouns are expected to be identified as important signals by faithful interpretations. We summarize how many times they are actually recognized as the top-k most important words by interpretation methods, where k is the interval varies in 10-20%, . . . , 90-100% of total tokens in an example. Results in Figure 4 demonstrate that interpretation methods from the same category have high correlations when extracting disambiguating nouns. For example, VaGrad and VaPGD leveraging gradients only, tend to position disambiguating nouns on the top of their importance lists, which is consistent with human judgments. Likewise, the perturbationbased methods, Occlusion and LIME, also put the disambiguation words to very similar positions.
On the Sensitivity and Stability of Model Interpretations in NLP
In our work, we enhanced the existed definition of interpretation faithfulness by two other notions and proposed corresponding quantitative metrics: sensitivity and stability, for each of them. We studied interpretations under the two notions along with the existed one. We found that interpretations have inconsistent performance regarding different criteria. We proposed a new class of interpretations, motivated by the adversarial robustness techniques, which achieves the best performance under the sensitivity and the stability criteria. We further proposed a novel paradigm to evaluate interpretations on the dependency parsing task, which moves beyond text classification in the literature. Our study shed light on understanding the behavior of model interpretations and suggested the community to put more efforts on defining an appropriate evaluation pipeline for interpretation faithfulness.
On the Sensitivity and Stability of Model Interpretations in NLP
This paper does not contain direct social influences. However, we believe the model analysis and interpretation techniques discussed in this paper are critical for deploying deep learning based models to real-world applications. Following previous work in this direction such as Jacovi and Goldberg (2020), we advocate to carefully consider the explanations obtained from interpretation methods as they may not always reflect the true reasoning process behind model predictions.
On the Sensitivity and Stability of Model Interpretations in NLP
Besides the three notions of faithfulness discussed in this paper, there are other important aspects for measuring interpretations that could be applied to evaluate interpretations. Also, We are not claiming that the proposed paradigm are perfect as faithfulness measurements. For example, we recognize that it requires further and detailed analysis on either the model itself or the interpretation methods lead to a low performance on the stability metric, although we do try to make sure models behaviors do not change substantially between an input pair.
On the Sensitivity and Stability of Model Interpretations in NLP
Moreover, experiments in this paper are all based on mainstream English corpora. Although our techniques are not language specific, there could be different conclusions given the varying properties of languages. For example, the discussion for dependency parsing could be easily affected by the language one considers. 
On the Sensitivity and Stability of Model Interpretations in NLP
Datasets Statistics of the datasets are presented in Table 5.   and Ba, 2015) with an initial learning rate of 1e-4, batch size 128, and no warm-up steps. We set the maximum number of fine-tuning to be 3. The finetuned BERT achieves 90.7, 95.4, and 96.9 accuracy on SST-2, Yelp and AGNews, respectively. When explaining BERT predictions, we only consider the contribution of word embeddings to the model output.
On the Sensitivity and Stability of Model Interpretations in NLP
For BiLSTM classifier, we use an one-layer BiL-STM encoder with a linear classifier. The embedding is initialized with the 100-dimensional pretrained GloVe word embedding. We use Adam with an initial learning rate of 1e-3, batch size 512, hidden size 100 and dropout rate 0.2 for training. We set the maximum number of epochs to be 20 but perform early stopping when the performance on the development set doesn't improve for three epochs. Our BiLSTM classifier receives 84.2, 93.3, 95.9 accuracy on SST-2, Yelp and AGNews, respectively.
On the Sensitivity and Stability of Model Interpretations in NLP
For DeepBiaffine, we simplify the original architecture by using a one-layer BiLSTM encoder and a biaffine classifier. The word embedding is also initialized with the pre-trained 100-dimensional GloVe word embedding while the part-of-speech tag embeddings are initialized to all zero. The encoder hidden size is 100. The arc and dependency relation hidden size are both 500. We get an UAS of 95.1 with our model. Note that for DeepBiaffine, each input token is represented by 1 https://pytorch.org/ the concatenation of its word embedding and its part-of-speech tag embedding. When applying the interpretation methods and the evaluation metrics, we only modify the word embeddings but keep the part-of-speech tag embeddings unchanged.
On the Sensitivity and Stability of Model Interpretations in NLP
For VaGrad, GradInp, VaPGD, PGDInp, and Ing-Grad, we use the automatic differentiation mechanism of PyTorch. For LIME, we modify the code from the original implementation of Ribeiro et al. (2016) 1 . For DeepLIFT, we use the implementation in Captum 2 . For Certify, we modify the code in auto_LiRPA 3 .
On the Sensitivity and Stability of Model Interpretations in NLP
For the two reference-based methods IngGrad and DeepLIFT, we use all zero word embeddings as the reference point. To approximate the integral in IngGrad, we sum up 50 points along the linear path from the reference point to the current point. For the perturbation-based methods LIME and Occlusion, we also set the word embedding of a token to an all zero embedding when it is perturbed.
On the Sensitivity and Stability of Model Interpretations in NLP
Hyper-parameter tuning For all interpretations that require hyper-parameter tuning, including LIME, PGDInp, VaPGD, we randomly select 50 examples from the development set and choose the best hyperparameters based on the performance on these 50 examples. Specifically, the number of perturbed examples around the original point for LIME to fit a linear regression model is selected from {100, 200, 500, 800}. For PGDInp and VaPGD, we select the best maximum perturbation norm ϵ as for BERT and BiLSTM classifier from {0.1, 0.5, 1.2, 2.2}. We set the number of iterations as 50, and the step size as ϵ/5. Note that we might be able to achieve better performance of VaPGD and PGDInp by also tuning the number of iterations and the step size. However, to keep the computational burden comparable with other interpretations, we do not tune these hyperparameters.
On the Sensitivity and Stability of Model Interpretations in NLP
Sensitivity Details We use PGD with a binary search for the minimal perturbation magnitude. In practice, we set the number of iterations to be 100 and the step size to be 1.0. Then, we conduct a binary search to estimate the smallest vicinity of the original point which contains an adversarial example that changes the model prediction.
On the Sensitivity and Stability of Model Interpretations in NLP
The synonyms in the stability metrics come from (Alzantot et al., 2018), where they extract nearest neighbors in the GloVe embeddings space and filter out antonyms with a counterfitting method. We allow at most four tokens replaced by their synonyms for each input and at most 0.1 change in the output probability of the model prediction for BERT and 0.2 for BiLSTM.
On the Sensitivity and Stability of Model Interpretations in NLP
Thresholds To compute the removal-based metrics and the AUC of sensitivity for text classification tasks, we vary the number of tokens being removed (preserved) or perturbed to be 10%, 20%, . . ., 50% of the total number of tokens in the input. For the dependency parsing task, the corresponding thresholds are 10%, 20% and 30%.
On the Sensitivity and Stability of Model Interpretations in NLP
We evaluate the interpretation methods under both the text classification task and the dependency parsing task. Below, we cover implementation details for each task, respectively, including what is the specific model score interpretation methods explain, and what metrics we use for that task.
On the Sensitivity and Stability of Model Interpretations in NLP
Text Classification Task s y (e (x)) is the probability after the Softmax function corresponding to the original model prediction. We apply all the metrics mentioned in the main paper: removalbased metrics, including comprehensiveness and sufficiency scores, sensitivity score, and stability score. For removal-based metrics, we replace the important tokens with the pad token as a proxy for removing it.
On the Sensitivity and Stability of Model Interpretations in NLP
Dependency Parsing Task s (e (x)) is the unlabeled arc log probability between the preposition and its head, i.e., unlabeled arc score after log_softmax, in the graph-based dependency parser. We discard the sufficiency score as it is unreasonable to remove a large proportion of tokens on a structured prediction task. We also discard the stability metric as there is little consensus on how to attack a structured model.
On the Sensitivity and Stability of Model Interpretations in NLP
We showcase an example for interpreting BERT and BiLSTM in Figure 5 and 
On the Sensitivity and Stability of Model Interpretations in NLP
It said analysts had been expecting a small profit for the period .
On the Sensitivity and Stability of Model Interpretations in NLP
It said analysts had been expecting a small profit for the period . Figure 7: An example of interpreting the PP attachment arc in the dependency parsing task. A deeper red color means the token is identified as more important for the model to predict the PP attachment arc.  
On the Sensitivity and Stability of Model Interpretations in NLP
We qualitatively study some cases where PGDInp does well under the removal-based criterion while GradInp does not. In Figure 9, we show an example from explaining BERT on the SST-2 dataset, with the importance scores given by PGDInp, VaPGD, GradInp, VaGrad and the comprehensiveness score. For PGDInp and GradInp, we show the exponential of importance scores.
On the Sensitivity and Stability of Model Interpretations in NLP
As shown in Figure 9, the importance score for each token given by GradInp is close to zero. Va-Grad also gives near zero importance scores. At the same time, PGDInp and VaPGD have distinguishable and meaningful importance scores.
On the Sensitivity and Stability of Model Interpretations in NLP
Based on the above observations, we suspect that the reason why PGD-based methods could avoid the failure of gradient-based methods is that they do not suffer from the gradient saturation issue. Gradient saturation refers to the cases where gradients are close to zero at some specific inputs and provide no information about the importance of different features of those inputs. Note that PGDbased methods consider not only a single input, but search on the vicinity of that input where the neighbors have none-zero gradients.
On the Sensitivity and Stability of Model Interpretations in NLP
However, notice that VaGrad works better than GradInp. We suspect that is because although all elements in the gradient vector are close to zero, the **L-2 norm** of it is still distinguishable. However, GradInp takes the **dot-product** between embeddings and their gradients as the importance score. It is likely that negative and positive dimensions are neutralized, making the importance scores undistinguishable, and thus the behavior of GradInp corrupted. This hypothesis needs further explorations and demonstrations.
On the Sensitivity and Stability of Model Interpretations in NLP
We use Student's t test to exam the superior performance of the best robustness-based methods against previous methods. The double checkmark represents a confidence level of 95% and a single checkmark for a confidence level of 90%.  
On the Sensitivity and Stability of Model Interpretations in NLP
Rank correlation = 0.689 Model change = 0.08 Original Supporters and rivals warn of possible fraud ; government says chavez 's defeat could produce turmoil in world oil market . Contrast Supporters and rivals warn of possible fraud ; government says chavez 's defeat could produce disorder in planet oil trade .
On the Sensitivity and Stability of Model Interpretations in NLP
Rank correlation = 0.317 Model change = 0.00 Original Mills corp. agreed to purchase a qqq percent interest in nine malls owned by general motors asset management corp. for just over qqq billion , creating a new joint venture between the groups . Contrast Mills corp. agree to purchase a qqq percent interest in nine malls owned by comprehensive motors asset management corp. for just over qqq trillion , creating a new joint venture between the groups .
On the Sensitivity and Stability of Model Interpretations in NLP
Rank correlation = 0.970 Model change = 0.12 Original London ( reuters ) -oil prices surged to a new high of qqq a barrel on wednesday after a new threat by rebel militia against iraqi oil facilities and as the united states said inflation had stayed in check despite rising energy costs . Contrast london ( reuters ) -oil prices surged to a new high of qqq a canon on wednesday after a new menace by rebel militia against iraqi oil facilities and as the united states said inflation had stayed in check despite rising energy costs . 
On the Sensitivity and Stability of Model Interpretations in NLP
Rank correlation = 0.633 Model change = 0.01 Original Last summer I had an appointment to get new tires and had to wait a super long time. I also went in this week for them to fix a minor problem with a tire they put on. They "fixed" it for free, and the very next morning I had the same issue. I called to complain, and the "manager" didn't even apologize!!! So frustrated. Never going back. They seem overpriced, too. Contrast Last summer I took an appoints to get new tires and had to wait a super long time. I also went in this week for them to fix a minor problem with a tire they put on. They "fixed" it for free, and the very impending morning I had the same issue. I called to complain, and the "manager" didn't even apologize!!! So frustrated. Never going back. They seem overpriced, too.    Table 11: Similarity between the parser outputs before and after applying the evaluation metric. We show that sensitivity changes the global model output less. Occlu. and DeepLI. represents Occlusion and DeepLIFT, respectively
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The increasing deployment of artificial intelligence (AI) systems in high stakes domains has been coupled with an increase in societal demands for these systems to provide explanations for their predictions. This societal demand has already resulted in new regulations requiring explanations [24,45,50,56]. These explanations can allow users to gain insight into the system's decision-making process, which is a key component in fostering trust and confidence in AI systems [21,55].
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
However, many machine learning techniques, which are responsible for much of the advances in AI, are not easily explainable, even by experts in the field. This has led to a growing research community [34], with a long history, focusing on "interpretable" or "explainable" machine learning techniques. 2 However, despite the growing volume of publications, there remains a gap between what society needs and what the research community is producing.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
One reason for this gap is a lack of a precise definition of an explanation. This has led to some ambiguity in regulations [24,45,50,56] and researchers defining their own problem statements, as well as solutions. A reason for the lack of precise definition is that different people in different settings may require different kinds of explanations. We refer to the people interacting with an AI system as consumers, and to their different types as personas [29]. For example, a doctor trying to understand an AI diagnosis of a patient may benefit from seeing known similar cases with the same diagnosis; a denied loan applicant will want to understand the main reasons for their rejection and what can be done to reverse the decision; a regulator, on the other hand, will want to understand the behavior of the system as a whole to ensure that it complies with the law; and a developer may want to understand where the model is more or less confident as a means of improving its performance.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Since there is no single approach to explainable AI that always works best, we require organizing principles for the space of possibilities and tools that bridge the gap from research to practice. In this paper, we provide a taxonomy and describe an open-source toolkit to address the overarching need, taking into account the points of view of the many possible explanation consumers. Our contributions are as follows:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Taxonomy Conception: we propose a simple yet comprehensive taxonomy of AI explainability that considers varied perspectives. This taxonomy is actionable in that it aids users in choosing an approach for a given application and may also reveal gaps in available explainability techniques. • Taxonomy Implementation: we architect an application programming interface and extensible toolkit that realizes the taxonomy in software. This effort is non-trivial given the diversity of methods. We have released the toolkit into the open source community under the name AI Explainability 360 (AIX360). It is the most comprehensive explainability toolkit across different ways of explaining (see Table 1 and Section 5). • Algorithmic Enhancements: we take several state-of-the-art interpretability methods from the literature and further develop them algorithmically to make them more appropriate and consumable in practical data science applications. • Educational Material: we develop demonstrations, tutorials, and other educational material to make the concepts of interpretability and explainability accessible to non-technical stakeholders. The tutorials cover several problem domains,  [1] ✓ Skater [7] ✓ ✓ ✓ H2O [4] ✓ ✓ ✓ InterpretML [6] ✓ ✓ ✓ EthicalML-XAI [3] ✓ DALEX [2] ✓ ✓ tf-explain [8] ✓ ✓ iNNvestigate [5] ✓ Table 1: Comparison of AI explainability toolkits.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
including lending, health care, and human capital management, and provide insights into their respective datasets and prediction tasks in addition to their more general educational value.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The current version of the toolkit contains eight explainability algorithms:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Boolean Decision Rules via Column Generation (BRCG) [17]:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Learns a small, interpretable Boolean rule in disjunctive normal form (DNF) for binary classification. • Generalized Linear Rule Models (GLRM) [59]: Learns a linear combination of conjunctions for real-valued regression through a generalized linear model (GLM) link function (e.g., identity, logit). • ProtoDash [26]: Selects diverse and representative samples that summarize a dataset or explain a test instance. Nonnegative importance weights are also learned for each of the selected samples. • ProfWeight [20]: Learns a reweighting of the training set based on a given interpretable model and a high-performing complex neural network. Retraining of the interpretable model on this reweighted training set is likely to improve the performance of the interpretable model. • Teaching Explanations for Decisions (TED) [30]: Learns a predictive model based not only on input-output labels but also on user-provided explanations. For an unseen test instance both a label and explanation are returned. • Contrastive Explanations Method (CEM) [18]: Generates a local explanation in terms of what is minimally sufficient to maintain the original classification, and also what should be necessarily absent.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Functions (CEM-MAF) [41]: For complex images, creates contrastive explanations like CEM above but based on high-level semantically meaningful attributes. • Disentangled Inferred Prior Variational Autoencoder (DIP-VAE) [38]: Learns high-level independent features from images that possibly have semantic interpretation.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The toolkit also includes two metrics from the explainability literature: Faithfulness [12] and Monotonicity [41]. The rest of the paper is organized as follows. In Section 2, we describe our taxonomy and show that it is simple to use and comprehensive. In Section 3, we describe its software implementation.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In Section 4, we outline enhancements made to the algorithms to make them more consumable, along with describing educational material aimed at making the toolkit more accessible to non-AI experts. In Section 5, we discuss other related toolkits and survey papers. In Section 6, we summarize our contributions and highlight promising directions along which the toolkit can be further enhanced and extended.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
As mentioned in the introduction, explainable artificial intelligence is a rapidly growing research field that has produced many varied methods. Even within the AIX360 toolkit, which represents a small fraction of this literature, different algorithms can cater to different types of consumers (e.g., data scientist, loan applicant) who may desire different kinds of explanations (e.g., feature-based, instancebased, language-based). It is important therefore to understand the diverse forms of explanations that are available and the questions that each can address. A proper structuring of the explanation space can have large bearing not just on researchers who design and build new algorithms, but perhaps more importantly on practitioners who want to understand which algorithms might be most suitable for their application and might be lost otherwise.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In Figure 1, we provide one such structuring of the explainability space. The structure is in the form of a small decision tree (which is itself considered to be interpretable [23]). Each node in the tree poses a question to the consumer about the type of explanation required. For the majority of consumers who are not experts in AI explainability, the AIX360 website provides a glossary to define terms used in the taxonomy and a guidance document for further context. Herein we note the following definitions:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• A static explanation refers to one that does not change in response to feedback from the consumer, while an interactive explanation allows consumers to drill down or ask for different types of explanations (e.g., through dialog) until they are satisfied. AIX360 focuses on static explanations, like much of the literature to date. • A local explanation is for a single prediction, whereas a global explanation describes the behavior of the entire model. • A directly interpretable model is one that by its intrinsic transparent nature is understandable by most consumers (e.g., a small decision tree), whereas a post-hoc explanation involves an auxiliary method to explain a model after it has been trained. One could also have a self-explaining model that itself generates local explanations, but may not necessarily be directly interpretable (e.g., rationale generation in text). • A surrogate model is a second, usually directly interpretable model that approximates a more complex model, while a visualization of a model may focus on parts of it and is not itself a full-fledged model.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The leaves in the decision tree are labeled with algorithms in the AIX360 toolkit or a question mark ('?') if the category is not represented in the current toolkit. We also depict the different modalities 3 for which each algorithm could be used.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Our primary intention here is to propose a taxonomy that is simple yet comprehensive enough so that it is useful for different types of users. We by no means claim it to be perfect or complete.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
We show that the taxonomy in Figure 1 allows different consumers to navigate the space of explanations and find the right algorithm or class of algorithms appropriate for their task. For simplicity, we narrate the scenario as if the consumers themselves were using the taxonomy, although the more common case is for a data scientist to make these decisions with the consumer's needs in mind.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
We consider a personal finance application where customers are applying for a loan from a financial institution such as a bank.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
As is typically the case in many deployed applications [19], we assume here that the bank has a complex existing process based on a combination of data-driven approaches and business rules to arrive at the final loan approval decision, which they would like to preserve as much as possible. One consumer persona working within this process is a loan officer, who may wish to validate the approval or denial recommendations of an AI model. Another persona clearly interested in the reasoning behind the decisions is an applicant whose loan was denied. They may want to know about a few factors that could be changed to improve their profile for possible approval in the future. A third persona who might be interested in the overall reasoning of the model is a data science executive. They may want some assurance that in most cases the recommendations made by the model are reasonable.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
We now look more closely at these three personas and show how our taxonomy can help them in choosing the right methods.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Loan Officer: As mentioned above, the loan officer is interested in validating whether recommendations given by the model for different loan applications are justified. One way for the loan officer to gain confidence in the recommendation made for an individual, whom we will call Alice and whose loan was approved, is to look at other individuals who were similar to Alice and were also approved. Using our taxonomy in Figure 1, they would first go down the "static" branch as a single explanation is desired. This would be followed by taking the "model", "local" and "post-hoc" branches as the loan officer wants explanations for particular individuals such as Alice based on an already built model. They would then choose a method under "samples" to find other individuals similar to Alice in the dataset. Given the current version of the AIX360 toolkit, they would then choose ProtoDash.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
If the training data for the model includes not only past loan decisions but also explanations written by past loan officers, then it is possible to give the current loan officer an explanation for Alice in the same form and using the same language. In this case, the loan officer would again traverse the hierarchy in similar fashion as before, except they would now choose the branch "direct" and pick TED if using the current version of AIX360. • Applicant: A different applicant, say Bob, may want to know why his loan was rejected and more importantly what he could do to change the decision in his favor in the future.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
To support Bob's desire, one would go down the "static," "model", "local", and "post-hoc" branches as he wants an explanation of the model's decision just for himself. One would then choose an appropriate method under "features" as Bob wants an explanation based on his profile. Given the current version of AIX360, CEM would be chosen to obtain such an explanation. • Bank Executive: Bank executives may not care so much about specific applicants as about the overall behavior of the model and whether it is, in general, basing its recommendations on robust justifiable factors. In this case, executives would follow the "static" and "model" branches followed by "global" as they want a high-level understanding of the model. Since the model is already deployed, they would then follow the "post-hoc" branch and finally choose "surrogate" to obtain an interpretation of the original model using another model they can understand. In this case, a natural choice from our toolkit would be ProfWeight. However, a directly interpretable method such as BRCG could also be used here where it is trained on the predictions of the original complex model. This follows the common strategy of model compression [14] or knowledge distillation [31].
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
We now show how some popular classes of explainability methods can be effectively categorized in our taxonomy. Again, we do not claim that this is a complete listing, although most of the methods reviewed by Guidotti et al. [25] are included.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Saliency Methods [static → model → local → post-hoc → features]: Saliency based methods [51,52], which highlight different portions in an image whose classification we want to understand, can be categorized as local explanation methods that are static and provide feature-based explanations in terms of highlighting pixels/superpixels. In fact, popular methods such as LIME [47] and SHAP [40] also fall under this umbrella. Counterfactual explanations [57], which are similar to contrastive explanations where one tries to find a minimal change that would alter the decision of the classifier, are another type of explanation in this category. 
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The AIX360 toolkit aims to provide a unified, flexible, and easy to use programming interface and an associated software architecture to accommodate the diversity of explainability techniques required by various stakeholders. The goal is to be amenable both to data scientist users, who may not be experts in explainability, as well as algorithm developers. Toward this end, we make use of a programming interface that is similar to popular Python model development tools (e.g., scikit-learn) and construct a hierarchy of Python classes corresponding to explainers for data, models, and predictions. Algorithm developers can inherit from a family of base class explainers in order to integrate new explainability algorithms. The organization of explainer classes takes into account not only the taxonomy described in Section 2 but also the perspective of the primary users of the toolkit, data scientists and developers. As such, it is not identical to the taxonomy in Figure 1 although there are clear correspondences. Instead, we have organized the base class explainers according to the AI modeling pipeline shown in Figure  2, based upon their use in offering explanations at different stages. Details are as follows:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Data explainers: These explainers are implemented using the base class DIExplainer (Directly Interpretable 4 unsupervised Explainer), which provides abstract methods for explainers that use unsupervised techniques to explain datasets. The AIX360 explainers that inherit from this class include Proto-dashExplainer and DIPVAEExplainer. • Directly interpretable explainers: These explainers are implemented using the base class DISExplainer (Directly Interpretable Supervised Explainer), which supports explainers that train interpretable models directly from labelled data. The AIX360 explainers that inherit from this class and implement its methods include BRCGExplainer and GLRMExplainer. Additionally, the TED_CartesianExplainer, which can train models based on data labelled with personaspecific explanations, also inherits from DISExplainer. Listing 1 shows a subset of abstract methods provided by DIS-Explainer and an example illustrating the use of BRCGExplainer. • Local post-hoc explainers: These explainers are further subdivided into black-box and white-box explainers. The blackbox explainers are model-agnostic and generally require access only to a model's prediction function while the whitebox explainers generally require access to a model's internals such as its loss function. Both classes of explainers are implemented in AIX360 via base classes LocalBBExplainer and LocalWBExplainer, respectively. The AIX360 explainers CEMExplainer and CEMMAFExplainer both inherit from LocalWBExplainer. Listing 1c shows an example of using the CEMExplainer to obtain pertinent negatives corresponding to MNIST images. • Global post-hoc explainers: These explainers are sub-divided into black-box and white-box explainers as above. The Glob-alWBExplainer base class includes abstract methods for explainers that train a more interpretable surrogate model given an original source model along with its data. The ProfweightExplainer in AIX360 inherits from this base class. In addition to the above base classes, AIX360 includes dataset classes to facilitate loading and processing of commonly used datasets so that users can easily experiment with the implemented algorithms.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Beyond the higher-level contributions discussed in Sections 2 and 3, AIX360 also includes enhancements to individual explainability methods as well as complementary components. These enhancements take methods as described in papers, which may only benefit machine learning researchers and expert data scientists, and make them accessible to a broader range of consumers. The enhancements presented in this section run the gamut from algorithmic to educational. For BRCG, a simplified algorithm was developed as detailed in Section 4.1 to avoid the need for an integer programming solver, which would be a major barrier to adoption. For TED, the barrier is the lack of datasets containing explanations along with labels. Section 4.2 discusses the synthesis of such a dataset based on rules for an employee retention application. In Sections 4.3 and 4.4, we discuss extensions that are built on top of existing model/algorithm outputs to improve their consumability: visualization for GLRM, and feature importance for ProtoDash and CEM. Section 4.5 documents the explainability metrics included in AIX360. Lastly, Section 4.6 describes an interactive web demo targeted at non-experts and featuring a credit approval use case approached from the perspectives of three types of consumers. Also discussed in Section 4.6 are tutorials aimed at data scientists looking to introduce explainability into different application domains.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Generation: "Light" version
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The BRCG algorithm produces a binary classifier in the form of a disjunctive normal form (DNF) rule (conjunctive normal form (CNF) is also possible through a simple transformation). The published version of BRCG [17] relies on the use of an integer programming solver such as CPLEX, which is often prohibitive due to cost or other barriers to access. In AIX360, we have replaced the integer programming components with a heuristic beam search algorithm and have modified the problem formulation accordingly. We call the resulting algorithm a "light" version of BRCG (BRCG-light).
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
To describe BRCG-light, we recall some notation from [17] (please see [17] for more context in general). Let n denote the training sample size and P and Z denote the sets of indices corresponding to positively and negatively labeled samples. Let K index the set of all possible (exponentially many) clauses that can be included in the DNF rule. These clauses are conjunctions of the features, which have been binarized as discussed in [17]. The binary-valued decision variable w k indicates whether clause k ∈ K is included in the rule, and c k is the complexity of the clause. For sample i, K i ⊆ K is the subset of clauses satisfied by the sample (which evaluate to true), and for i ∈ P, ξ i indicates a false negative error.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
BRCG-light attempts to solve the following problem related to [17, eq. (1)-(4)]:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Like in [17], the first two terms of the objective function in (1) represent the Hamming loss of the classifier. Different from [17], the last term in the objective is a penalty on the complexity of the rule, whereas [17] uses an upper bound on complexity. Herein the clause complexities c k are parametrized by a fixed cost λ 0 plus a variable cost λ 1 multiplying the number of conditions (literals) in the clause (see the objective function in (2) below), whereas [17] only considered the equivalent of λ 1 = λ 0 . These modifications bring BRCG-light closer to GLRM [59] in two respects: 1) They conform to the GLRM problem formulation in which rule complexity is penalized in the same way, giving AIX360 users a common interface; 2) They allow reuse of a GLRM sub-algorithm as explained below. A minor difference from [17] is that the first two terms in the objective in (1) are scaled by 1/n to facilitate the setting of λ 0 and λ 1 across datasets of different sizes. BRCG-light uses the same column generation approach as in [17] to effectively handle the exponential number of variables in (1). Below we summarize the steps in this approach:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
(1) Solve the linear programming (LP) relaxation of (1), obtained by relaxing the w k ∈ {0, 1} constraint to w k ≥ 0, restricted to a small subset of clauses S ⊂ K in place of K.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
(2) Solve a pricing problem given below in (2) to find clauses omitted from step 1 that can improve the current LP solution and add them to S. The pricing problem for BRCG-light is also slightly modified from that in [17]. Define µ i , i ∈ P to be the optimal dual variables associated with the constraints ξ i + k ∈K i w k ≥ 1 in (1), whose values are determined by solving the LP in step 1 above. Denote by x i j , i = 1, . . . , n, j = 1, . . . , d, the value of the ith sample of the jth binarized feature, and letx i j = 1 − x i j . Clauses are encoded by a binary-valued vector z with components z j indicating whether the jth feature participates in the clause. The BRCG-light pricing problem is as follows:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In [17], steps 2 and 4 of the column generation procedure are performed by calling the integer programming solver CPLEX. For BRCG-light, we avoid integer programming in step 2 by observing instead that (2) is an instance of the column generation subproblem for GLRM [59, eq. (9)], with the following identifications: r i = 1 for i ∈ Z (corresponding to set I + in [59]), r i = −nµ i for i ∈ P (corresponding to set I − ), and a = δ . As a consequence, we may use the heuristic beam search algorithm described in [59] to obtain a good feasible solution to (2). The constraint j z j ≤ D in (2) can be met by setting the maximum degree (i.e. depth) of the beam search to D. Since the column generation procedure requires only a solution to (2) with negative objective value to improve the LP solution from step 1, and not necessarily an optimal solution to (2), the substitution of a heuristic algorithm for integer programming has less of an effect. On the other hand, if the beam search algorithm fails to find a solution to (2) with negative objective value, then we do not have a certificate that such a solution does not exist, unlike with integer programming run to completion.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
BRCG-light also avoids integer programming in step 4. For this, we observe that (1) (with or without the restriction to subset S) is in fact an instance of a slight generalization of the GLRM column generation subproblem [59, eq. (9)]. The identifications are as follows: r i = 1 for i ∈ P corresponding to set I + , an empty set I − = ∅, a = ξ , λ 0 = 0, indices k ∈ S instead of j = {1, . . . , d }, and a binary variable b ik indicating whether the ith sample satisfies the kth clause in place ofx i j . The slight generalization is that the sum λ 1 d j=1 z j in [59, eq. (9)] is replaced by the weighted sum k ∈S λ k z k with weights λ k = c k + (1/n) i ∈Z b ik . It follows that a slight generalization of the beam search algorithm in [59] can produce good solutions to (1) restricted to S, as required in step 4.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In terms of implementation in AIX360, the beam search algorithm from [59] has been generalized as discussed above and made available to both BRCG-light and GLRM as a common Python module. The modules implementing GLRM follow [59] plus the enhancement of Section 4.3 below while the module implementing BRCG-light is novel.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In Appendix A, we present trade-off curves between classification accuracy and rule complexity for BRCG-light in comparison to the integer programming version of BRCG (BRCG-IP) as well as previous rule learning methods evaluated in [17]. These comparisons show that BRCG-light occupies an intermediate position, inferior to BRCG-IP as may be expected, but superior to the earlier methods. In particular, a beam search width of 5 (the default in AIX360) gives better results than smaller widths on most datasets. Larger widths might give still better results with increased computational complexity.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The TED framework [30] requires a training dataset that includes explanations (E) in addition to features (X ) and labels (Y ). These explanations are used to teach the framework what are appropriate explanations for a given feature set in the same manner that a typical training dataset teaches what are appropriate labels (Y ). To address this gap between the needs of the framework (datasets with explanations) and current practice (datasets with no explanations), we devised a general strategy for creating synthetic datasets, modeled after the approach described in [30] for predicting tic-tac-toe moves.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Consider the use case of determining which employees should be targeted with a retention action to dissuade them from leaving the company [53]. Ideally, we would ask a human resource (HR) specialist with deep knowledge of the circumstances for employee retention to develop a training dataset where each employee, represented as a feature vector, would have a prediction of whether the employee was at risk to leave, justified with an explanation.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Since we do not have such information, we simulate this process by encoding the HR specialist's knowledge into a set of rules. Each rule is a function with a feature vector as its inputs and the prediction of whether the employee is a retention risk (Y ) as its output. We use the rule number as the explanation for this prediction since it should be meaningful to the target persona.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Given these rules, we can then generate a synthetic dataset of any size by first generating a random collection of feature vectors and then applying the rules to determine the appropriate predictions and explanations. AIX360 contains such a dataset for employee retention with 10,000 entries, created by an available python script that can be modified by changing the distribution of features, changing the rules, or the size of the dataset. It can also serve as a example for the generation of datasets for other classification problems.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In summary, to use this approach for a classification problem, one must determine the following:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
(1) the features of the problem (2) the distribution of each feature to be generated and any dependence among features (3) the rules for mapping feature vectors to predictions; the rule number will be used as the explanation For the employee retention dataset included with AIX360, we used 8 features (Position, Organization, Potential, Rating, Rating Slope, Salary Competitiveness, Tenure Duration, and Position Duration) and various distributions for each feature, resulting in a feature space of over 80 million possibilities. We created 25 rules, motivated by common retention scenarios, such as not getting a promotion in a while, not being paid competitively, receiving a disappointing evaluation, being a new employee in organizations with inherently high attrition, not having a salary that is consistent with positive evaluations, mid-career crisis, etc. In the current implementation of this use case, we further vary the application of these rules depending on various positions and organizations. For example, an organization may have much higher attrition because their skills are more transferable outside the company.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
This dataset reinforces two advantages of the TED framework:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• the explanations match the terminology of the domain; they represent higher-level semantic concepts, e.g., "not being paid competitively", created by the domain expert, rather than mathematical combinations of feature values, • a single explanation "not being paid competitively" can occur in multiple ways in the feature space. "Not being paid competitively" rules will likely have different flavors depending on one's organization, position, and tenure. Although this will map to multiple rules, the explanation provided to the consumer can be a single concept.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The GLRM algorithm produces linear combinations of conjunctive 'AND' rules, which provide predictions directly in the case of linear regression or through a transformation in the case of generalized linear models (e.g., logistic regression). GLRM models can therefore be interpreted like linear models and their constituent rules are also readily understood individually. Nevertheless, the discussion in [59,Sec. 6] as well as our experience in putting together the AIX360 web demo and tutorials (Section 4.6) raise the question of the degree to which a GLRM model with (say) tens of rules is interpretable. While interpretation is still feasible, it may require a non-trivial investment of effort. The authors of [59] suggest a decomposition of GLRM models to reduce the effort of interpretation. Since first-degree rules, i.e. those that place a single condition on one feature, do not create interactions between features, they collectively form a type of generalized additive model (GAM) [27], i.e. a sum of univariate functions f j of individual features x j ,
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The same is true of linear functions of original, unbinarized numerical features (so-called "linear terms") if they are included in the GLRM model, so these can be added to the GAM. The benefit of having a GAM is that it can be visualized as a series of plots of the functions f j , thus summarizing the effects of all first-degree rules and linear terms. Higher-degree rules, which do involve feature interactions, can then be studied separately.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In AIX360, we have made the above suggestion a reality by equipping GLRM models with the ability to visualize the GAM part of the model. This is implemented as a visualize() method of the GLRM Python classes. Given a trained GLRM object and for each original feature x j , the visualize() method collects the associated first-degree rules and linear terms along with their model coefficients. It also collects information needed for plotting such as the domain of x j and its mean and standard deviation if they were removed during standardization. In addition, the method computes the importance of each feature to the GAM (though not necessarily to the overall GLRM), which is used to order the plots and possibly for other purposes. Importance is measured by the variance var(f j (X j )) with respect to the empirical distribution of X j . This accounts for both the variation of f j as well as that of X j (a near-constant function f j or near-constant values for X j would both yield low variance). Lastly, the explain() method of the GLRM classes, which prints a list of rules and linear terms, has the option of printing only higher-degree rules.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The base algorithms for both ProtoDash [26] and CEM [18] do not return feature importances. What they return are vectors that are the same dimension as the input. This might be sufficient for an expert data scientist to use to interpret, however, it might still pose a challenge to a less experienced user. In the AIX360 toolkit we add another layer of interpretability by also discerning features that are important in more easily interpreting the output. This is clearly observed in the tutorials that showcase the utility of these methods.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In particular, let x denote the original input with x [j] denoting its j th dimension. For ProtoDash when used to explain an input, let x ′ denote a prototype found by it for x. Then we compute the feature importance for feature j relative to x ′ denoted by θ j , where σ j denotes the standard deviation of feature j as follows:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Here |·| denotes absolute value. The above equation creates a feature importance value in [0, 1]. The higher the value more similar the feature and hence higher its importance. The top θ j s could inform the user of which and how many features of the prototype x ′ are close to that of x making it a good exemplar based explanation for it. For CEM given an input x classified by a classifier in class y we output two vectors, namely, a pertinent positive (PP) vector x pp and a pertinent negative (PN) vector x pn . A PP informs us of what minimal values for different features not exceeding those in x would be sufficient to obtain the same prediction y. While, a PN informs us of what minimal features if added or increased in value would change the classification of x to something other than y. Given these definitions we compute the feature importances for the j th feature for PPs and PNs as follows:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The higher the corresponding θ j values, the more important that feature is to either maintain or change the class depending on if its a PP or a PN respectively. The feature importances thus provide insight to the user on which factors are possibly most critical in the classification, which we expose now through the toolkit.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Although explanations are difficult to quantitatively evaluate and the gold standard is typically human evaluation [21], the AIX360 toolkit provides two quantitative metrics that serve as proxies of how "good" a particular feature-based local explanation is likely to be.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The first metric we implement is a version of the Faithfulness metric [12], which tries to evaluate if the feature importances returned by an explainability method are the correct ones. The method starts by replacing the most important feature value by another value termed as the base value which the user supplies as being a no information or no-op value. For grey-scaled images the base value would be zero for pixels, indicating that it is blank. Once this replacement has been done the new example is passed to the classifier and its prediction probability for the original predicted class is noted. The most important feature value is then restored and the second most important features value is replaced by the base value and again the prediction probability is noted. This process continues for all features that we have feature importance for. Given a feature importance vector θ and the corresponding prediction probabilities p, with ρ denoting Pearsons correlation we compute the following metric:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The higher the ϕ the better the explanation. The intuition is that removing more important features should reduce the confidence of the classifier more than removing less important ones. The second metric is Monotonicity [41], which simply tries to measure if adding more positive evidence increases the probability of classification in the specified class. In this case we start with the base value vector and keep incrementally replacing the features that have positive relevance with their actual values in increasing order of importance and note the classification probabilities in each case. If the features are (independent) positively correlated with the decision, then the class probabilities should monotonically increase.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Although both of these metrics are not novel in themselves, we are possibly the first explainability toolkit that has quantitative metrics to measure the "goodness" of explanations, as is indicated in Table 1.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
AIX360 was developed with the goal of providing accessible resources on explainability to non-technical stakeholders. Therefore, we include multiple educational materials to both introduce the explainability algorithms provided by AIX360, and to demonstrate how different explainability methods can be applied in real-world scenarios. These educational materials include a general guidance for the key concepts of explainability, a web demo that illustrates the usage of different explainability methods, and multiple tutorials.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The web demo was created based on the FICO Explainable Machine Learning Challenge dataset [22], a real-world scenario where a machine learning system is used to support decisions on loan applications by predicting the repayment risk of the applicants. Sample screenshots from the demo are shown in Figure 6. The demo highlights that three groups of people -data scientists, loan officers, and bank customers -are involved in the scenario and their needs are best served by different explainability methods. For example, although the data scientist may demand a global understanding of model behavior through an interpretable model, which can be provided by the GLRM algorithm, a bank customer would ask for justification for their loan application results, which can be generated by the CEM algorithm. We use storytelling and visual illustrations to guide users of AIX360 through these scenarios of different explainability consumers.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The AIX360 toolkit currently includes five tutorials in the form of Jupyter notebooks that show data scientists and other developers how to use different explanation methods across several application domains. The tutorials thus serve as an educational tool and potential gateway to AI explainability for practitioners in these domains. The tutorials cover:
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Using three different methods to explain a credit approval model to three types of consumers, based on the FICO Explainable Machine Learning Challenge dataset [22]. • Creating directly interpretable healthcare cost prediction models for a care management scenario using Medical Expenditure Panel Survey data [10]. • Exploring dermoscopic image datasets used to train machine learning models that help physicians diagnose skin diseases.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Understanding National Health and Nutrition Examination Survey datasets to support research in epidemiology and health policy. • Explaining predictions of a model that recommends employees for retention actions from a synthesized human resources dataset.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Beyond illustrating the application of different methods, the tutorials also provide considerable insight into the datasets that are used and, to the extent that these insights generalize, into the respective problem domains. These insights are a natural consequence of using explainable machine learning and could be of independent interest.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In this section, we review existing work that relates specifically to the two main topics of this paper: taxonomies and toolkits for explainable AI. We do not focus on the much larger literature on explainability methods themselves, although Section 2.2 does connect many of these to our taxonomy. The rapid expansion of explainable AI research has motivated several recent papers that survey the field and identify criteria to systematically classify explainability methods and techniques [9,15,25,43]. For example, Adadi and Berrada [9] conduct an extensive literature review and classify explainability methods according to three criteria: 1) The complexity or interpretability of the model to be explained, i.e., whether it is directly interpretable (intrinsic), a category that we also have in Figure 1, or a black-box model that requires post-hoc methods, also represented in Figure 1; 2) The scope of interpretability, corresponding to our local/global split in Figure 1; and 3) The dependency on the model used, i.e., whether the explanation technique applies to any type of machine learning model (general) or to only one type (model specific). Besides these three criteria, Carvalho et al. [15] add a criterion on the stage of model development where an explainability method is applied, i.e., whether it is before (pre-model), during (in-model), or after (post-model) building the machine learning model. As discussed in Section 3, the software implementation of our taxonomy is organized around a similar consideration. Lastly, several recent papers advocate for the vision of interactive explanation [32,37,42,61], allowing users to drill down or ask for different types of explanation (e.g., through dialogues) until satisfied with their understanding. In Figure 1 we include static/interactive as the first split, and note that there is a scarcity of available techniques for interactive explanations. The proposed taxonomy ( Figure 1) and software architecture (Section 3) add to the criteria above by making both broader distinctions (data vs. model) as well as finer distinctions (features vs. samples, black-box vs. white-box post-hoc).
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Although the above criteria provide a useful understanding of the interpretability literature, they by themselves do not imply actionability in terms of guiding a practitioner in selecting explainability methods. A recent paper by Guidotti et al. [25] attempts to create a detailed mapping of popular explainability methods (explanator) such as feature importance, saliency masks, and prototype selection to the kinds of AI explanation problems faced. These problems include obtaining directly interpretable surrogate models and "black-box explanation", the latter of which is further divided into the problems of model explanation, outcome explanation, and model inspection. The paper further provides a list of algorithms to generate each type of explanation. Although more actionable, this paper is limited to the consideration of post-hoc explanation of black-box models and ignores other aspects such as understanding data [9,32].
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
We have argued in Sections 1 and 2 for the necessity of mapping explainability methods to explanation problems because AI is being used in diverse contexts by diverse groups of people, who may have different needs in demanding more transparent AI. Several works capture these diverse motivations for explainability. For example, Doshi-Velez and Kim [21] enumerate the needs for explainability including to gain scientific understanding, ensure safety in complex tasks, guard against discrimination, and identify mismatched objectives. Adadi and Berrada [9] summarize the goals of AI explainability methods as justifying a model's decisions to users, improving user control of the AI by avoiding errors, enabling users to better improve the AI, and helping people discover new knowledge.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In addition to survey papers proposing taxonomies, there are multiple explainability toolkits available through open source. Some of the more popular ones are shown in Table 1 along with the categories of methods they possess. The clearest difference between AIX360 and these other toolkits is that AIX360 offers a wider spectrum of explainability methods, notably data explanations, metrics, and persona-specific explanations, the last referring to TED's ability to provide explanations in the language of the persona. Alibi [1] primarily possesses local explainability methods such as Anchors [48], contrastive explanations [18] and counterfactuals [57]. Skater [7] has directly interpretable models such as rule-based models and decision trees. These can also be used for global post-hoc interpretation. It also has LIME-like [47] methods for local post-hoc explainability. H2O [4] and InterpretML [6] both are similar to Skater in that they have methods that provide direct, local and global post-hoc explainability. The difference is in the particular methods they possess under these categories. Both EthicalML-XAI [3] and DALEX [2] have different ways to visualize model behavior in terms of input variables, which could provide insight into global behavior. DALEX also has standard local explainability methods such as LIME and SHAP. tf-explain [8] and iNNvestigate [5] primarily contain methods that provide local explanations for images. tf-explain also has ways to visualize activations of neurons of a neural network which could aid in understanding its global behavior.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Beyond differences in methods offered, AIX360 also distinguishes itself from other toolkits in terms of educational materials. In particular, AIX360 has materials aimed at less technical consumers, notably the web demo described in Section 4.6 which appears to be unique, as well as the guidance and glossary noted in Section 2. AIX360 also targets industry-specific users with the tutorials and web demo in Section 4.6. Another distinctive feature of AIX360 is its common architecture and programming interface.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
In summary, we have introduced the open-source AI Explainability 360 (AIX360) toolkit, which contains eight state-of-the-art explainability algorithms that can explain an AI model or a complex dataset in different ways to a diverse set of users. Some of the algorithms were enhanced from their published versions for AIX360 to make them more usable and their outputs easier to consume. The toolkit also contains two explainability metrics, a credit approval demonstration, five elaborate tutorials covering different real-world use cases, and 13 Jupyter notebooks, making it accessible to practitioners and non-experts. These resources, together with the breadth of the toolkit and its common extensible programming interface, help distinguish AIX360 from existing explainability toolkits.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
We have also provided a simple yet comprehensive taxonomy that structures the explainability space and serves multiple purposes, including guiding practitioners, revealing algorithmic gaps to researchers, and informing design choices related to explainability software (e.g., class hierarchy) for data scientists and developers. We thus believe that our toolkit and taxonomy provide technical, educational, and operational benefits to the community over and above what currently exists. As AI surges forward with trust being one of the critical bottlenecks in its widespread adoption [46], we hope that the community will leverage and significantly enhance our contributions.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
The AIX360 toolkit solicits contributions related to explainable AI to enable it to grow. This includes the categories discussed in Section 3, where extensions can be easily handled. Of particular interest are contributions to categories that are present in the taxonomy of Figure 1 but not in the toolkit. These are indicated in Figure  1 by "?" in three cases: interactive, static → data → distributions, and static → model → global → post-hoc → visualize. There is a significant literature in the 'visualize' category, which includes visualizing intermediate nodes and/or layers in a deep neural network [44,60] as well as plotting the effects of input features on the output [43] to assess their importance. Such contributions would be highly relevant for the toolkit. There is comparatively much less work in the other two categories, notwithstanding calls for interactive explanation [32,42,61], and we hope that the taxonomy inspires more research in those directions. Another area where contributions would be highly welcome are for modalities that are not covered under the categories that do have algorithms (e.g., contrastive for text). Contributions in categories that are missing from the taxonomy in Figure 1 are of course valuable as well.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Lastly, a large variety of deep learning frameworks are available today to train and store models (e.g., TensorFlow, Keras, PyTorch, MXNet). Therefore explainability toolkits should allow users to explain models that have been built using different frameworks while avoiding the need to implement explainability algorithms multiple times for each framework. To ease the task of algorithm developers and make explainability algorithms framework-independent, future work should implement framework-specific model classes that expose a common API needed by explainability algorithm developers. Preliminary work in this direction was initiated in AIX360 by defining a class for Keras-based classifiers that is utilized by CEMExplainer and CEMMAFImageExplainer.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
We would like to thank MaryJo Fitzgerald for assistance with the web demo, Karthikeyan Natesan Ramamurthy for assistance with the Medical Expenditure Panel Survey data, and Joanna Brewer for copy editing.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
A ACCURACY-COMPLEXITY TRADE-OFFS FOR BRCG-LIGHT  show trade-offs between classification accuracy and DNF rule complexity for BRCG-light (described in Section 4.1 and labelled BRCG-l in the figure legends) in comparison to the integer programming version of BRCG (BRCG-IP), Bayesian Rule Sets (BRS) [58], and the alternating minimization (AM) and block coordinate descent (BCD) algorithms from [54]. For BRCG-light, three settings of the beam search width are evaluated: B = 1, 3, 5. The 15 datasets considered here are the same as in [17]. Complexity is measured as the number of clauses in the DNF rule (equivalently the number of rules in the rule set) plus the total number of conditions in all clauses/rules, also following [17]. Overall, the comparisons show that BRCG-light provides trade-offs that are worse on average than BRCG-IP, as may be expected given the dramatic reduction in computation, but better than the other methods. They also show that B = 5 gives better results than smaller widths on most datasets, particularly the larger ones. 
On the Lack of Robust Interpretability of Neural Text Classifiers
In recent years, large scale language models like BERT and RoBERTa have helped achieve new state-of-the-art performance on a variety of NLP tasks (Devlin et al., 2019;Liu et al., 2019). While relying on vast amounts of training data and model capacity has helped increase their accuracy, the reasoning of these models is often hard to comprehend. To this end, several techniques have been proposed to interpret the model predictions.
On the Lack of Robust Interpretability of Neural Text Classifiers
Perhaps the most widely-adopted class of interpretability approaches is that of feature-based in- * Work done during internship at Amazon. terpretability where the goal is to assign an importance score to each of the input features. These scores are also called feature attributions. Several methods in this class (e.g., SHAP (Lundberg and Lee, 2017), Integrated Gradients (Sundararajan et al., 2017)) possess desirable theoretical properties making them attractive candidates for interpretability.
On the Lack of Robust Interpretability of Neural Text Classifiers
Benchmarking analyses often show that these methods possess high fidelity, i.e., removing features marked important by the interpretability method from the input indeed leads to significant change in the model output as expected (Atanasova et al., 2020;Lundberg and Lee, 2017).
On the Lack of Robust Interpretability of Neural Text Classifiers
However, relatively few investigations have been carried out to understand the robustness of feature attributions. To explore the robustness, we conduct two tests based on randomization: Different Initializations Test: This test operationalizes the implementation invariance property of Sundararajan et al. (2017). Given an input, it compares the feature attributions between two models that are identical in every aspect-that is, trained with same architecture, with same data, and same learning schedule-except for their randomly chosen initial parameters. If the predictions generated by these two models are also identical, one would also expect the feature attributions to be the same for such functionally equivalent models. If the attributions in two cases are not the same, two users examining the same input may deem the same features to have different importance based on the model that they are consulting. Untrained Model Test: This test is similar to the test of Adebayo et al. (2018). Given an input, it compares the feature attributions generated on a fully trained model with those on a randomly initialized untrained model. The test evaluates whether feature attributions on a fully trained model differ from the feature attributions computed on an untrained model as one would expect.
On the Lack of Robust Interpretability of Neural Text Classifiers
We conduct the two tests on a variety of text classification datasets. We quantify the feature attribution similarity using interpretation infidelity (Arras et al., 2016) and Jaccard similarity (Tanimoto, 1958). The results suggest that: (i) Interpretability methods fail the different initializations test. In other words, two functionally equivalent models lead to different ranking of feature attributions; (ii) Interpretability methods fail the untrained model test, i.e., the fidelity of the interpretability method on an untrained model is better than that of random feature attributions.
On the Lack of Robust Interpretability of Neural Text Classifiers
These findings may have important implications for how the prediction interpretations are shown to the users of the model, and raise interesting questions about reliance on these interpretations. For instance, if two functionally equivalent models generate different interpretations, to what extent can a user act upon them, e.g., investing in a financial product or not. We discuss these implications and potential reasons for this behavior in §4.
On the Lack of Robust Interpretability of Neural Text Classifiers
Related work. Model interpretability has different aspects: local (e.g. Lundberg and Lee, 2017) vs. global (e.g. Tan et al., 2018), feature-based (e.g. Lundberg and Lee, 2017) vs. concept-based (e.g.  vs. hidden representationbased (Li et al., 2016). See Gilpin et al. (2018); Guidotti et al. (2018) for an overview. In this paper, we focus on feature-based interpretability, which is a well-studied and commonly used form (Bhatt et al., 2020;Tjoa and Guan, 2020). Specifically, this class consists of a relatively large number of methods, of which some have been shown to satisfy desirable theoretical properties (e.g., SHAP (Lundberg and Lee, 2017), Integrated Gradients (Sundararajan et al., 2017), and DeepLIFT (Shrikumar et al., 2017)).
On the Lack of Robust Interpretability of Neural Text Classifiers
There are several important aspects of interpretation robustness. Some prior studies have considered interpretability in the context of adversarial robustness where the goal often is to actively fool the model to generate misleading feature attributions. See for instance Anders et al. (2020) Slack et al. (2020). In this work, rather than focusing on targeted changes in the input or the model, we explore robustness of feature attribution methods to various kinds of randomizations.
On the Lack of Robust Interpretability of Neural Text Classifiers
Several prior works have focused on quantifying quality of interpretations. See for instance, Ade-  Yang and Kim (2019). Closest to ours is the work of Adebayo et al. (2018), which is based on checking the saliency maps of randomly initialized image classification models. However, in contrast to Adebayo et al., we consider text classification. Moreover, while the analysis of Adebayo et al. is largely based on visual inspection, we extend it by considering automatically quantifiable measures. We also extend the analysis to non-gradient based methods (SHAP).
On the Lack of Robust Interpretability of Neural Text Classifiers
We describe the datasets, models, and interpretability methods considered in our analysis. Datasets. We consider four different datasets covering a range of document lengths and number of label classes. The datasets are: (i) FPB: The Financial Phrase Bank dataset (Malo et al., 2014) where the task is to classify news headlines into one of three sentiment classes, namely, positive, negative, and neutral. (ii) SST2: The Stanford Sentiment Treebank 2 dataset (Socher et al., 2013). The task is to classify single sentences extracted from movie reviews into positive or negative sentiment classes. (iii) IMDB: The IMDB movie reviews dataset (Maas et al., 2011). The task is to classify movie reviews into positive or negative sentiment classes. (iv) Bios: The Bios dataset of De-Arteaga et al. (2019). The task is to classify the profession of a person from their biography. Table 5 in Appendix A shows detailed dataset statistics.
On the Lack of Robust Interpretability of Neural Text Classifiers
We consider four pretrained Transformer encoders: BERT (BT), RoBERTa (RB), DistilBERT (dBT), and DistilRoBERTa (dRB). The encoder is followed by a pooling layer to combine individual token embeddings, and a classification head. Appendix B.1 describes the detailed architecture, training and hyperparameter tuning details. After training and hyperparameter tuning, the best model is selected based on validation accuracy and is referred to as Init#1. Different Initializations Test. Recall from §1 that this test involves comparing two identical models trained from different initializations. The second model, henceforth referred to as Init#2, is trained using the same architecture, hyperparameters and training strategy as Init#1, but starting from a different set of initial parameters. Since we start from pretrained encoders, the encoder parameters are not intialized. For each layer in the rest of the model, a set of initial parameters different from those in Init#1 is obtained by calling the parameter initialization method of choice for this layer-He initialization (He et al., 2015) in this case-but with a different random seed. Untrained Model Test. Recall that this test involves comparing the trained model (Init#1) with a randomly initialized untrained model, henceforth called Untrained. To obtain Untrained, we start from the Init#1, and randomly initialize the fully connected layers attached on top of the Transformer encoders (the encoder weights are not randomized). The initialization strategy is the same as in Different Initializations Test.
On the Lack of Robust Interpretability of Neural Text Classifiers
We consider a mix of gradient-based and model agnostic methods.  (2017). We also include random feature attribution (RND) which corresponds to each feature being assigned an attribution from the uniform distribution, U(0, 1). Appendix B.2 provides details about the parameters chosen for the interpretability methods.
On the Lack of Robust Interpretability of Neural Text Classifiers
Given an input text document, we tokenize the text using the tokenizer of the corresponding encoder. Finally, for each input feature (that is, token), the feature attribution of the gradient-based methods is a vector of the same length as the token input embedding. For scalarizing these vector scores, we use the L2-norm strategy of Arras et al. (2016) and the Input Gradient strategy of Ding et al. (2019).
On the Lack of Robust Interpretability of Neural Text Classifiers
To compare the feature attributions by various interpretability methods, we use the following metrics.
On the Lack of Robust Interpretability of Neural Text Classifiers
(In)Fidelity: Given an input text which has been split into L tokens, t = [t 1 , . . . , t L ], get the vector Ψ(t) = [ψ(t 1 ), . . . , ψ(t L )] of feature attributions of the corresponding tokens using the interpretability method to be evaluated. Drop the features from t in the decreasing order of attribution score until the model prediction changes from the original prediction (with all tokens present). Infidelity is defined as the % of features that need to be dropped until the prediction changes. A better interpretability method is expected to need a lower fraction of features to be dropped until the prediction change. We simulate feature dropping by replacing the corresponding input token with the model's unknown vocabulary token.
On the Lack of Robust Interpretability of Neural Text Classifiers
The infidelity metric has appeared in many closely related forms in a number of studies evaluating model interpretability (Arras et al., 2016;Atanasova et al., 2020;DeYoung et al., 2020;Fong et al., 2019;Lundberg and Lee, 2017;Samek et al., 2017). All of these forms operate by iteratively hiding features in the order of their importance and measuring the change in the model output, e.g., in predicted class probability, or the predicted label itself. We chose number of tokens to prediction change, which is closely aligned with (Arras et al., 2016), due to its simplicity as compared to more involved metrics relying on AUC-style measures (Atanasova et al., 2020;Samek et al., 2017).
On the Lack of Robust Interpretability of Neural Text Classifiers
Jaccard Similarity: It is common to show top few most important features to users as model interpretations. See for instance, Ribeiro et al. (2016) and Schmidt and Biessmann (2019). In order to measure the similarity between feature attributions generated by different methods, we use the Jac-card@K% metric. Given an input t, let s i be the set of top-K% tokens, when the tokens are ranked based on their importance as specified by an attribution output Ψ i . Then, given two attribution outputs Ψ i and Ψ j , Jaccard@K% measures the similarity between them as: J(i, j) = |s i ∩s j | |s i ∪s j | . If the top-K% tokens by the two attributions Ψ i and Ψ j are the same, then J(i, j) = 1. In case of no overlap in the top-K% tokens, J(i, j) = 0.
On the Lack of Robust Interpretability of Neural Text Classifiers
Appendix D shows some examples of Jac-card@K% computation.  68  64  61  59  50  54  53  54  46  53  52  50  28  22  27  25  SG  68  63  61  58  46  50  53  51  45  52  52  48  27  23  27  23  IG  68  62  61  59  38  48  50  49  39  47  46  47  24  20  17  22  SHP  60  54  52  43  22  25  30  27  10  19  11  13  15  15  15  14  RND  72  71  68  70  61  67  66  69  58  58  63  66  51  51 49 56  Table 2 shows the infidelity of different interpretability methods on the best performing models (Init#1). The table shows that: (i) As expected, the infidelity of all interpretability methods is better than RND; (ii) SHP provides the best performance, followed by IG; (iii) For a given dataset, even though different encoders have very similar accuracy (Table 1), the infidelity of the same interpretability method for different encoders can vary widely, e.g., SHP on IMDB; (iv) There is no particularly discernable correlation between the models and their infidelity, for instance, with FPB dataset, the distilled Transformers provides same or lower infidelity as compared to the original counterparts (BERT, RoBERTa), whereas the trend is reversed for SST2 data. Moreover, gradient-based methods in Table 2 use the L2-norm reduction ( §2.2). Table 8 in Appendix C.2 shows that in most cases, the performance is much worse when using the Input Gradient reduction. Hence, for the rest of the analysis, we only use L2-norm reduction.
On the Lack of Robust Interpretability of Neural Text Classifiers
Comparing Init#1 and Init#2 in Table 6 in Appendix C.1-two otherwise identical models with only difference being the random initial parameters, shows that a vast majority of predictions are common between the two models: meaning that the two models are almost functionally equivalent.
On the Lack of Robust Interpretability of Neural Text Classifiers
We now compare the similarity in feature attributions of two functionally equivalent models. Since the feature attributions are generated w.r.t. the predicted class, our similarity analysis is limited to samples where both models generate the same prediction. Figure 1 shows Jaccard@25% when comparing the feature attributions of Init#1 vs.  Init#2, and Init#1 vs. Untrained. The figure shows that (i) for the functionally equivalent models Init#1 vs. Init#2, Jaccard@25% is far from the ideal value of 1.0 -in fact, for IG, the value drops to almost 0.5; (ii) When comparing the top-ranked feature attributions of Init#1 vs. Init#2, and Init#1 vs. Untrained, the former should show a much bigger overlap than latter, but this is not the case, except for SHP. Tables 3 and 4 show the results for rest of the cases, revealing similar insights. Specifically, the Jaccard@25% between Init#1 and Untrained for VN averaged over all 16 cases (four datasets, four models in Table 3) is 68, whereas the same is 69 when comparing Init#1 and Untrained (Table 4). The same comparison yields 67 vs. 64 for SG, 56 vs. 50 for IG and 65 vs. 29 for SHP. Moving beyond averages, we also counted for each of the cases in Table 3, the number of times Jaccard@25% between Init#1 and Init#2 is within 10 units (Jaccard@25% ranges from 0-100) of the Jaccard@25% between Init#1 and Untrained for the corresponding  65  52  77  67  63  53  73  63  72  47  78  72  76  70  81  75  SG  53  66  70  62  68  63  70  66  57  45  76  75  75  70  81  78  IG  46  35  68  51  53  37  71  50  65  36  82  68  49  51  73  68  SHP  57  55  64  55  63  65  66  62  68  50  75  67  72  71  75  69   Table 3: Jaccard@25% between the feature attributions for Init#1 vs. Init#2 models (shown as %).
On the Lack of Robust Interpretability of Neural Text Classifiers
FPB SST2 IMDB Bios BT RB dBT dRB BT RB dBT dRB BT RB dBT dRB BT RB dBT dRB In other words, the attribution overlap between two functionally equivalent models can be similar to that between a trained vs. an untrained model. Figure 2 shows the infidelity of different methods on SST2 dataset with a BT model. We note that the performance of RND is better (lower infidelity) for the untrained model (Untrained) than for the trained model (Init#1). Furthermore, even for the untrained model (Untrained), all interpretability methods have a better fidelity than RND. In fact, for SHP, the infidelity is almost half of RND. Table 9 in Appendix C shows a similar pattern for the rest of the datasets and models.
On the Lack of Robust Interpretability of Neural Text Classifiers
In short, even for an untrained model, the interpretability methods lead to better-than-randomattribution fidelity. The insights highlight the need for baselining the fidelity metric with untrained models before using it as an evaluation measure.
On the Lack of Robust Interpretability of Neural Text Classifiers
We carried out two tests to assess robustness of several popular interpretability methods on Transformer-based text classifiers. The results show that both gradient-based and model-agnostic methods can fail the tests.
On the Lack of Robust Interpretability of Neural Text Classifiers
These observations raise several interesting questions: if the fidelity of the interpretations is reasonably high on even an untrained model, to what extent does the interpretability method reflect the data-specific vs. data-independent behavior of the model? If two functionally equivalent models lead to different feature attributions, to what extent can the practitioners rely upon these interpretations to make consequential decisions? One cause of the non-robust behavior could be the redundancy in text where several input tokens may provide evidence for the same class (e.g., several words in input review praising the movie). Another reason, related to the first, could be the pathologies of neural models where dropping most of the input features could still lead to highly confident predictions (Feng et al., 2018). 1 Dropping individual features can also lead to out-of-distribution samples, further limiting the effectiveness of methods and metrics that rely on simulating feature removal (Kumar et al., 2020;Sundararajan and Najmi, 2020). Systematically analyzing the root causes, and designing interpretability measures that are cognizant of the specific characteristics of text data-preferably with human involvement (Chang et al., 2009;Doshi-Velez and Kim, 2017;Hase and Bansal, 2020;Nguyen, 2018;Poursabzi-Sangdeh et al., 2021;Schmidt and Biessmann, 2019)-is a promising research direction. Similarly, extending the Untrained Model Test to study the effect of randomization of pre-trained embedding models on interpretability is another direction for exploration.  We insert a classification head on top of the pretrained encoder. The end-to-end classifier has the following architecture: Encoder → Avg. pooling → FC-layer (512-units) → RELU → FC-layer (K units), where K is the number of classes. The maximum sequence length of the encoder is set to 128 for FPB and SST2 datasets, 512 for IMDB reviews and 200 for the Bios data. Each dataset is split into a 80% − 20% train-test set. 10% of the training set is used as a validation set for hyperparameter optimization. Accuracy and overlap statistics are reported on the test set.
On the Lack of Robust Interpretability of Neural Text Classifiers
We used the following hyperparameter ranges: learning rate {10 −2 , 10 −3 , 10 −4 , 10 −5 } and the number of last encoder layers to be fine-tuned {0, 2}. Fine tuning last few layers of the encoder, as opposed to all the layers, has been shown to lead to superior test set performance (Sun et al., 2019).
On the Lack of Robust Interpretability of Neural Text Classifiers
We use the AdamW optimizer (Loshchilov and Hutter, 2019). The maximum number of training epochs is 25. We use early stopping with a patience of 5 epochs: if the validation accuracy does not increase for 5 consecutive epochs, we stop the training. The model training was done using Py-Torch (Paszke et al., 2019) and HuggingFace Transformers (Wolf et al., 2020) libraries.
On the Lack of Robust Interpretability of Neural Text Classifiers
Owing to the large runtime of methods like SHAP and Integrated Gradients, interpretations are only computed for a randomly chosen 1000 subsample from the test set. Consequently, metrics like fidelity and Jaccard@K% are reported only on this subset.   Vanilla Saliency and SmoothGrad are implemented using the PyTorch autograd function. Integrated Gradients and SHAP are implemented using Captum (Kokhlikyan et al., 2020). The parameters of these methods are:  (Sundararajan et al., 2017) that has the same dimensionality as the model input, but consists of 'non-informative' feature values. We construct the baseline by computing the embedding value of the unknown vocabulary token and repeating it N times where N is the maximum sequence length of the model. KernelSHAP. Requires two parameters. (i) Number of feature coalitions: Following the author implementation, 2 we use a value of 2L + 2 11 , where L is the number of input tokens in the text. (ii) Dropped token value: SHAP operates by dropping subsets of tokens and estimating model output on these perturbed inputs. We simulate dropping of a 2 https://github.com/slundberg/shap token by replacing its embedding value with that of the unknown vocabulary token. Table 6 shows the fraction of predictions common between Init#1 and Init#2. Table 7 shows the accuracy of Untrained. As expected, the accuracy of Untrained is much smaller than with trained models. Table 8 shows the infidelity of gradient-based interpretability methods when using the Input Gradient dot product reduction of Ding et al. (2019). When comparing the results to those with L2 reduction in Table 2, we notice that in all except two cases (VN and SG on dBT with IMDB data), the performance is worse. Table 9 shows the infidelity for the Untrained model. Much like Figure 2, the table shows that in several cases, the performance of the feature attribution methods (most notably SHP) can be much better than random attribution (RND).
On the Lack of Robust Interpretability of Neural Text Classifiers
The table also shows an exception for dBT on IMDB dataset where for all methods, the infidelity is near 100. This behavior is likely an artefact of the particular initial parameters due to which the model always predicts a certain class irrespective of the input.
On the Lack of Robust Interpretability of Neural Text Classifiers
We now show some examples of Jaccard@K% computation. The examples show the input text, different models, and top-K% tokens ranked w.r.t. their importance. The attribution method used was VN.
On the Lack of Robust Interpretability of Neural Text Classifiers
Example 1: SST2 data. Comparing Init#1 and Init#2. Both models predict the sentiment to be positive.
On the Lack of Robust Interpretability of Neural Text Classifiers
Text. at heart the movie is a deftly wrought suspense yarn whose richer shadings work as coloring rather than substance Top-25% w.r.t. Init#1. {'substance', 'rather', 'at', 'yarn', 'coloring', 'movie'} Top-25% w.r.t. Init#2. {'heart', '##tly', 'suspense', 'at', 'yarn', 'def'} Jaccard@25%. 20
On the Lack of Robust Interpretability of Neural Text Classifiers
Example 2: SST2 data. Comparing Init#1 and Init#2. Both models predict the sentiment to be positive.
On the Lack of Robust Interpretability of Neural Text Classifiers
Text. an infectious cultural fable with a tasty balance of family drama and frenetic comedy Top-25% w.r.t. Init#1. {'fable', 'infectious', 'cultural', 'balance', 'an'} Top-25% w.r.t. Init#2. {'cultural', 'balance', 'infectious', 'fable', 'an'} Jaccard@25%. 100
On the Lack of Robust Interpretability of Neural Text Classifiers
Example 3: FPB data. Comparing Init#1 and Untrained. Both models predict the sentiment to be negative.
On the Lack of Robust Interpretability of Neural Text Classifiers
Text. nokia shares hit 13.21 euros on friday , down 50 percent from the start of the year in part because of the slow introduction of touch-screen models Top-25% w.r.t. Init#1. { ', ','.','down','friday','shares','euros','nokia',','.','down','euros','friday','hit','shares','nokia' } Jaccard@25%. 100 In second and third examples, even though the rankings are different, the set of top-25% tokens is the same leading to a perfect Jaccard@25%.