What are some common techniques for gradient-based feature attribution in neural networks?
Neural networks are often viewed as black
boxes because their inner workings are too
complicated for a user to understand why the
model produced a particular prediction for a given
input. This shortcoming has spawned an active
field of research in developing methods to better
understand and explain how neural networks work.
For example, feature attribution methods aim to
measure the sensitivity of a model’s predictions
to the values of individual input features. Many
of these methods quantify feature attribution as
the gradient of the model’s output with respect
to an input feature (Simonyan et al., 2013;
Smilkov et al., 2017; Sundararajan et al., 2017).
We note that while the general reliability and
faithfulness of gradient-based methods has been
a contentious area of research (Adebayo et al.,
2018; Yona and Greenfeld, 2021; Amorim et al.,
2023), gradient-based methods have nonetheless
continued to be widely used (Han et al., 2020;
Supekar et al., 2022; Novakovsky et al., 2022)
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting important words in the corresponding input text. While this might be useful for tasks where decisions are explicitly influenced by individual tokens in the input, we suspect that such highlighting is not suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers. Influence functions explain the decisions of a model by identifying influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which ‘saliency maps’ may not have clear interpretation. Furthermore, we develop a new quantitative measure based on influence functions that can reveal artifacts in training data.
[{"first":"Xiaochuang","last":"Han","middle":[]},{"first":"Byron C.","last":"Wallace","middle":[]},{"first":"Yulia","last":"Tsvetkov","middle":[]}]
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Background Autism spectrum disorder (ASD) is a highly heterogeneous disorder that affects nearly 1 in 189 females and 1 in 42 males. However, the neurobiological basis of gender differences in ASD is poorly understood, as most studies have neglected females and used methods ill-suited to capture such differences. Aims To identify robust functional brain organisation markers that distinguish between females and males with ASD and predict symptom severity. Method We leveraged multiple neuroimaging cohorts (ASD n = 773) and developed a novel spatiotemporal deep neural network (stDNN), which uses spatiotemporal convolution on functional magnetic resonance imaging data to distinguish between groups. Results stDNN achieved consistently high classification accuracy in distinguishing between females and males with ASD. Notably, stDNN trained to distinguish between females and males with ASD could not distinguish between neurotypical females and males, suggesting that there are gender differences in the functional brain organisation in ASD that differ from normative gender differences. Brain features associated with motor, language and visuospatial attentional systems reliably distinguished between females and males with ASD. Crucially, these results were observed in a large multisite cohort and replicated in a fully independent cohort. Furthermore, brain features associated with the motor network's primary motor cortex node predicted the severity of restricted/repetitive behaviours in females but not in males with ASD. Conclusions Our replicable findings reveal that the brains of females and males with ASD are functionally organised differently, contributing to their clinical symptoms in distinct ways. They inform the development of gender-specific diagnoses and treatment strategies for ASD, and ultimately advance precision psychiatry.
[{"first":"Kaustubh","last":"Supekar","middle":[]},{"first":"Carlo","last":"de los Angeles","middle":[]},{"first":"Srikanth","last":"Ryali","middle":[]},{"first":"Kaidi","last":"Cao","middle":[]},{"first":"Tengyu","last":"Ma","middle":[]},{"first":"Vinod","last":"Menon","middle":[]}]
Axiomatic Attribution for Deep Networks
We study the problem of attributing the prediction of a deep network to its input features, a problem previously studied by several other works. We identify two fundamental axioms---Sensitivity and Implementation Invariance that attribution methods ought to satisfy. We show that they are not satisfied by most known attribution methods, which we consider to be a fundamental weakness of those methods. We use the axioms to guide the design of a new attribution method called Integrated Gradients. Our method requires no modification to the original network and is extremely simple to implement; it just needs a few calls to the standard gradient operator. We apply this method to a couple of image models, a couple of text models and a chemistry model, demonstrating its ability to debug networks, to extract rules from a deep network, and to enable users to engage with models better.
[{"first":"Mukund","last":"Sundararajan","middle":[]},{"first":"Ankur","last":"Taly","middle":[]},{"first":"Qiqi","last":"Yan","middle":[]}]
Obtaining genetics insights from deep learning via explainable artificial intelligence
Artificial intelligence (AI) models based on deep learning now represent the state of the art for making functional predictions in genomics research. However, the underlying basis on which predictive models make such predictions is often unknown. For genomics researchers, this missing explanatory information would frequently be of greater value than the predictions themselves, as it can enable new insights into genetic processes. We review progress in the emerging area of explainable AI (xAI), a field with the potential to empower life science researchers to gain mechanistic insights into complex deep learning models. We discuss and categorize approaches for model interpretation, including an intuitive understanding of how each approach works and their underlying assumptions and limitations in the context of typical high-throughput biological datasets. In this Review, the authors describe advances in deep learning approaches in genomics, whereby researchers are moving beyond the typical ‘black box’ nature of models to obtain biological insights through explainable artificial intelligence (xAI).
[{"first":"Gherman","last":"Novakovsky","middle":[]},{"first":"Nick","last":"Dexter","middle":[]},{"first":"Maxwell","last":"Libbrecht","middle":["W."]},{"first":"Wyeth","last":"Wasserman","middle":["W."]},{"first":"Sara","last":"Mostafavi","middle":[]}]
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
This paper addresses the visualisation of image classification models, learnt using deep Convolutional Networks (ConvNets). We consider two visualisation techniques, based on computing the gradient of the class score with respect to the input image. The first one generates an image, which maximises the class score [Erhan et al., 2009], thus visualising the notion of the class, captured by a ConvNet. The second technique computes a class saliency map, specific to a given image and class. We show that such maps can be employed for weakly supervised object segmentation using classification ConvNets. Finally, we establish the connection between the gradient-based ConvNet visualisation methods and deconvolutional networks [Zeiler et al., 2013].
[{"first":"Karen","last":"Simonyan","middle":[]},{"first":"Andrea","last":"Vedaldi","middle":[]},{"first":"Andrew","last":"Zisserman","middle":[]}]
SmoothGrad: removing noise by adding noise
Explaining the output of a deep network remains a challenge. In the case of an image classifier, one type of explanation is to identify pixels that strongly influence the final decision. A starting point for this strategy is the gradient of the class score function with respect to the input image. This gradient can be interpreted as a sensitivity map, and there are several techniques that elaborate on this basic idea. This paper makes two contributions: it introduces SmoothGrad, a simple method that can help visually sharpen gradient-based sensitivity maps, and it discusses lessons in the visualization of these maps. We publish the code for our experiments and a website with our results.
[{"first":"Daniel","last":"Smilkov","middle":[]},{"first":"Nikhil","last":"Thorat","middle":[]},{"first":"Been","last":"Kim","middle":[]},{"first":"Fernanda","last":"Vi'egas","middle":[]},{"first":"Martin","last":"Wattenberg","middle":[]}]
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Deep learning models have become increasingly complex, and unfortunately their inscrutability has grown in tandem with their predictive power (Doshi-Velez and Kim, 2017). This has motivated efforts to design example-specific approaches to interpreting black box NLP model predictions, i.e., indicating specific input tokens as being particularly influential for a given prediction. This in turn facilitates the construction of saliency maps over texts, in which words are highlighted with intensity proportional to continuous 'importance' scores. Prominent examples of the latter include gradient-based attribution (Simonyan et al., 2014;Sundararajan et al., 2017;Smilkov et al., 2017), LIME (Ribeiro et al., 2016), and attention-based (Xu et al., 2015) heatmaps.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
While widely used and potentially useful for some lexicon-driven tasks (e.g., sentiment analysis), we argue that by virtue of being constrained to highlighting individual input tokens, saliency maps will necessarily fail to explain predictions in more complex semantic tasks involving reasoning, such as natural language inference (NLI), where fine-grained interactions between multiple words or spans are key (Camburu et al., 2018). Moreover, saliency maps are inherently limited as a model debugging tool; they may tell us which inputs the model found to be important, but not why.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
To address these shortcomings, we investigate the use of what Lipton (2018) referred to as explanation by example. Instead of constructing importance scores over the input texts on which the model makes predictions, such methods rank training examples by their influence on the model's prediction for the test input (Caruana et al., 1999;Koh and Liang, 2017;Card et al., 2019). Specifically, we are interested in the use of influence functions (Koh and Liang, 2017), which are in a sense inherently 'faithful' in that they reveal the training examples most responsible for particular predictions. These do not require any modifications to the model structure.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
This paper presents a series of experiments intended to evaluate the potential utility of influence functions for better understanding modern neural NLP models. In this context, our contributions in-… Figure 1: A sentiment analysis example interpreted by gradient-based saliency maps (left) and influence functions (right). Note that this example is classified incorrectly by the model. Positive saliency tokens and highly influential examples may suggest why the model makes the wrong decision; tokens and examples with negative saliency or influence scores may decrease the model's confidence in making that decision. clude answering the following research questions. RQ1 We empirically assess whether the approximation to the influence functions (Koh and Liang, 2017) can be reliably used to interpret decisions of deep transformer-based models such as BERT (Devlin et al., 2019). RQ2 We investigate the degree to which results from the influence function are consistent with insights gleaned from gradient-based saliency scores for representative NLP tasks. RQ3 We explore the application of influence functions as a mechanism to reveal artifacts (or confounds) in training data that might be exploited by models.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
To the best of our knowledge, this is the first work in NLP to compare interpretation methods that construct saliency maps over inputs with methods that explain predictions via influential training examples. We also propose a new quantitative measurement for the effect of hypothesized artifacts ( 
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
As a standard, illustrative 'explanation-by-inputfeatures' method, we focus on gradient-based saliency maps, in which the gradient of the loss L is computed with respect to each token t in the input text, and the magnitude of the gradient serves as a feature importance score (Simonyan et al., 2014;Li et al., 2016a). Gradients have the advantage of being locally 'faithful' by construction: they tell us how much the loss would change, were we to perturb a token by a small amount. Gradient-based attributions are also agnostic with respect to the model, as long as it is differentiable with respect to inputs. Finally, calculating gradients is computationally efficient, especially compared to methods that require post-hoc input perturbation and function fitting, like LIME (Ribeiro et al., 2016). We are interested in why the model made a particular prediction. We therefore define a loss Lŷ with respect to the predictionŷ i that the model actually made, rather than the ground truth y i . For each token t ∈ x i , we define a saliency score −∇ e(t) Lŷ · e(t), where e(t) is the embedding of t. This is also referred as the "gradient × input" method in Shrikumar et al. (2017). The "gradient" ∇ e(t) Lŷ captures the sensitivity of the loss to the change in the input embedding, and the "input" e(t) leverages the sign and magnitude of the input. The final saliency score of each token t would be L1-normalized across all tokens in x i .
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Unlike Simonyan et al. (2014) and Li et al. (2016a), when scoring features for importance, we do not take the absolute value of the saliency score, as this encodes whether a token is positively influencing the prediction (i.e., providing support the prediction) or negatively influencing the prediction (highlighting counter-evidence). We show an example in the left part of Figure 1.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
In contrast to explanations in the form of tokenlevel heatmaps, the influence function provides a method for tracing model predictions back to training examples. It first approximates how upweighting a particular training example (x i , y i ) in the training set {(x 1 , y 1 ), . . . , (x n , y n )} by i would change the learned model parametersθ:
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We can then use the chain rule to measure how this change in the model parameters would in turn affect the loss of the test input (as in saliency maps, w.r.t. the model prediction):
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
More details (including proofs) can be found in Koh and Liang (2017).
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We define the influence score for each training example (x i , y i ) as − dLŷ d i , and then z-normalize it across all examples in the training set. Note that since Lŷ is defined with respect to a particular test input, influence scores of training examples are also defined for individual test instances.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Intuitively, a positive influence score for a training example means: were we to remove this example from the train set, we would expect a drop in the model's confidence when making the prediction on the test input. A negative influence score means that removing the training example would increase the model's confidence in this prediction. We show an example in the right part of Figure 1.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We are interested in analyzing and comparing the two interpretation approaches (gradient-based attributions and influence functions) on relatively shallow, lexicon-driven tasks and on more complex, reasoning-driven tasks. We focus on sentiment analysis and natural language inference (NLI) as illustrative examples of these properties, respectively. Both models are implemented on top of BERT encoders (Devlin et al., 2019). In particular we use BERT-Base, with the first 8 of the 12 layers frozen, only fine-tuning the last 4 transformer layers and the final projection layer. 3 It is worth noting that influence functions are guaranteed to be accurate only when the model is strictly convex (i.e., its Hessian is positive definite and thus invertible) and is trained to convergence. However, deep neural models like BERT are not convex, and one often performs early stopping during training. We refer to Koh and Liang (2017) for details on how influence functions can nonetheless provide good approximations. To summarize briefly: for the non-convexity issue, we add an appropriate 'damping' term to the model's Hessian so that it is positive definite and invertible. Concerning non-convergence: the approximated influence may still be interpretable as the true influence of each training example plus a constant offset that does not depend on the individual examples. Aside from this theory, we also perform a sanity check in §4 to show that influence functions can be applied to BERT in practice on the two tasks that we consider.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We use a binarized version of the Stanford Sentiment Treebank (SST-2) (Socher et al., 2013). Our BERT-based model is trained on 10k examples; this achieves 89.6% accuracy on the SST-2 dev set of 872 examples. We randomly sample 50 examples from the SST-2 dev set as the set for which we extract explanations for model predictions.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Natural language inference Our deeper 'semantic' task is NLI, a classification problem that concerns the relationship between a premise sentence and a hypothesis sentence. NLI is a ternary task with three types of premise-hypothesis relations: entailment, neutral, and contradiction. We train our BERT model on the Multi-Genre NLI (MNLI) dataset (Williams et al., 2018), which contains 393k premise and hypothesis pairs of three relations from 10 different genres. We collapse the neutral and contradiction labels to a single non-entailment label and only use 10k randomly sampled examples for training. On the MNLI dev set of 9815 examples, the model achieves an accuracy of 84.6%.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
To evaluate model interpretations in a controlled manner, we adopt a diagnostic dataset, HANS (Mc-Coy et al., 2019). This contains a balanced number of examples where hypotheses may or may not entail premises with certain artifacts that they call 'heuristics' (e.g., lexical overlap, subsequence Influence functions are designed to be an approximation to leave-one-out training for each training example. But the theory only proves that this works on strictly convex models. While Koh and Liang (2017) show that influence functions can be a good approximation even when the convexity assumption is not satisfied (in their case, a CNN for image classification), it is still not obvious that the influence function would work for BERT. Therefore, we conduct a sanity check: for each instance in our test set, we by turns remove the most positively influential 10%, the most negatively influential 10%, the least influential (where influence scores are near zero) 10%, and a random 10% of training examples. We are interested in how these removals in retraining would affect the confidence of model predictions. Table 1 and Table 2 show the result of experiments on sentiment analysis and NLI, repeated with 5 random initialization seeds.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Avg. ∆ in prediction confidence  it is reasonable to ask if they seem to tell similar stories regarding specific predictions. We propose two experiments that aim to estimate the consistency between these two interpretation methods. The first experiment addresses whether a token with high saliency also appears more frequently in the training examples that have relatively high influence. For each example in the test set, we find the tokens with the most positive, most negative, and median saliency scores. We then find all the influential training examples w.r.t. the test inputs that contain one of these tokens. These training examples could have any labels in the label set. We further only consider examples whose label is the same as the test prediction, because the token saliency scores, whether positive or negative, are directly w.r.t. the test prediction, and the effect of a token in an oppositely labeled training example is therefore indirect.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We compute the average influence score of these training examples and report the results on top 10%, 20%, 50%, and all training examples for both sentiment analysis and NLI tasks in Figure 2 and Figure 3 respectively. The reason we have results at different granularity is that from empirical results in Koh and Liang (2017), we see that the influence function approximations tend to be less accurate when going from the most influential to the less influential examples down in the spectrum.  In the task of sentiment analysis, we observe that training examples containing the most positively salient token in the test example generally have a higher influence to the test prediction. However, we do not see this trend (in fact, it is the opposite) in the task of natural language inference.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
The second experiment answers the question of whether the influence result would change significantly when a salient token is removed from the   input. Again, for each of the test examples, we identify the tokens with the most positive, most negative, and median saliency score. We by turns remove them from the input and compute the influence distribution over all training examples. We compare these new influence results with the one on the original input, and report an overlap rate of the top 0.1%, 0.2%, 0.5%, and 1% influential training examples before and after the token removal. Table 3 and Table 4 show results for sentiment analysis and NLI, respectively. When removing a token with the most positive saliency score, we expect the model to be less confident about its current prediction; it could possibly make a different prediction. Therefore, we expect to see a most different influence distribution from the original influence result compared to removing the token with median or the most negative saliency score. This is exactly what we observe in Table 3 for sentiment analysis. However, for NLI, we again see a rather opposite trend: removing the most negatively salient token (might make the prediction more confident but should not change the prediction itself) leads to the most different influence distribution.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We conclude from the above two experiments that gradient-based saliency maps and influential examples are compatible and consistent with each other in sentiment analysis. However, for NLI the two approaches do not agree with each other and could potentially tell very different stories. To this end, we take a closer look at the task of NLI.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Are saliency-based explanations useful for NLI? Gradient-based saliency maps are faithful by construction, but this does not mean that they will highlight input tokens that humans find plausible or useful. We hypothesize that highlighting individual input tokens as important is likely most useful for 'shallow' classification tasks like sentiment analysis, and less so for more complex reasoning tasks such as NLI.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
To contrast the types of explanations these methods offer in this context, we show explanations for a prediction made for a typical example in HANS in the form of a saliency map and influential examples in Table 5. The tokens that get the most positive and most negative saliency scores are marked in cyan and red, respectively. The training examples with the most positive and most negative influence scores are presented as supporting and opposing instances, respectively.  [entail] P: I don't know if I was in heaven or hell, said Lillian Carter, the president's mother, after a visit. H: The president's mother visited.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
[entail] P: Inverse price caps. H: Inward caps on price.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
[entail] P: Do it now, think 'bout it later. H: Don't think about it now, just do it.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
[entail]  The relationship classification decision in NLI is often made through an interaction between multiple words or spans. Therefore, an importance measure on each individual token might not give us much useful insight into model prediction. Though influence functions also do not explicitly tell us which latent interactions between words or spans informed the model prediction, we can test whether the model is relying on some hypothesized artifacts in a post-hoc way by looking at patterns in the influential training examples.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
In Table 5, though the most influential examples (both supporting and opposing) are ostensibly far from the test input, they all exhibit lexical overlap between the premise and hypothesis. Some of the influential training examples (e.g., the 4th supporting example and 2nd opposing example) capture a reverse ordering of spans in the premise and hypothesis. We note that our test input also has a high lexical overlap and similar reverse ordering. This exposes a problem: the model might be relying on the wrong artifacts like word overlap during the decision process rather than learning the relationship between the active and passive voice in our case. This problem was surfaced by finding influential examples. Because of this imbalance in prediction, they conclude that the models are perhaps exploiting artifacts in data when making decisions.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We see one potential problem out of the above method: it can only be applied to a certain group of examples and imply a general model behavior by examining the prediction imbalance. However, model behavior should depend on the actual example it sees each time. The extent to which the model exploits the artifact in each individual example remains unclear.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
To analyze the effect of artifacts on individual examples, we propose a method using influence functions. We hypothesize that if an artifact in-forms the model's predictions for a test instance, the most influential training examples for this test example should contain occurrences of said artifact. For instance, if our model exploits 'lexical overlap' when predicting the relation between a premise and a hypothesis, we should expect the most influential training examples found by the influence function to have a highly overlapping premise and hypothesis.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
In Figure 4a, we plot each training example's influence score and lexical overlap rate between its premise and hypothesis for a typical example in the HANS dataset. In linen with our expectation, the most influential (both positively and negatively) training examples tend to have a higher lexical overlap rate. Note that we also expect this trend for the most negatively influential examples, because they influence the model's prediction as much as the positively influential examples do, only in a different direction.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
To quantify this bi-polarizing effect, we find it natural to fit a quadratic regression to the influenceartifact distribution. We would expect a high positive quadratic coefficient if the artifact feature appears more in the most influential examples. For an irrelevant feature, we would expect this coefficient to be zero. With this new quantitative measure, we are ready to explore the below problems unanswered by the original diagnostic dataset.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
For test examples predicted as non-entailment, did the model fail to recognize that they have a lexical overlap feature? Was the artifact not exploited in these cases? Figure 4a and Figure 4b show two examples in HANS, one predicted as entailment and the other predicted as non-entailment. We observe that the example predicted as nonentailment does not have a significantly different influence-artifact pattern from the entailment example. In fact, the average quadratic coefficients for all examples predicted as entailment and nonentailment are +3.28 × 10 −3 and +3.30 × 10 −3 respectively. Therefore, for predicted non-entailment examples, we still see that the most influential training examples tend to have a high rate of lexical overlap, indicating that the model still recognizes the artifact in these cases.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
The model relies on training examples with high lexical overlap when predicting in the artificial HANS dataset. Would it still exploit the same artifact for natural examples? Apart from finding the most influential training examples for each HANS example, we also apply influence functions on 50 natural MNLI examples, not controlled to exhibit any specific artifacts. A typical example is shown in Figure 4c. The average quadratic coefficient over all 50 natural examples is +0.65 × 10 −3 , which is considerably smaller than the above cases in HANS dataset. The model therefore does not rely on as much lexical overlap in natural examples as in the diagnostic dataset.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We have been analyzing scenarios focusing on one data artifact. What if we have a second artifact during prediction possibly indicating a contradicting decision? How will the model recognize the two artifacts in such a scenario? We know that lexical overlap could be a data artifact exploited by NLI models for making an entailment prediction in HANS. On the other hand, as briefly pointed out by McCoy et al. (2019), other artifacts like negation might be indicative of non-entailment. We are interested in how two contradicting artifacts might compete when they both appear in an example. We take all examples in HANS labeled as entailment and manually negate the hypothesis so that the relationship becomes non-entailment. For example, a hypothesis "the lawyers saw the professor" would become "the lawyers did not see the professor".  Table 6. We observe that in the original HANS example, negation is actually a negative artifact: the training examples with negation tend to be the least influential ones. In the negated HANS example, we see the effect of negations becomes positive, while the effect of lexical overlap is drastically weakened. This confirms that the model recognizes the new set of artifacts, and the two are competing with each other.
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
Importantly, observing an artifact in the most influential training examples is a necessary but not sufficient condition to concluding that it was truly exploited by the model. However, it can serve as a first step towards identifying artifacts in blackbox neural models and may be complemented by probing a larger set of hypothesized artifacts.    
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We compared two complementary interpretation methods-gradient-based saliency maps and influence functions-in two text classification tasks: sentiment analysis and NLI. We first validated the reliability of influence functions when used with deep transformer-based models. We found that in a lexicon-driven sentiment analysis task, saliency maps and influence functions are largely consistent with each other. They are not consistent, however, on the task of NLI. We posit that influence functions may be a more suitable approach to interpreting models for such relatively complex natural language 'understanding' tasks (while simpler attribution methods like gradients may be sufficient for tasks like sentiment analysis).
Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
We introduced a new potential use of influence functions: revealing and quantifying the effect of data artifacts on model predictions, which have been shown to be very common in NLI. Future work might explore how rankings induced over training instances by influence functions can be systematically analyzed in a stand-alone manner (rather than in comparison with interpretations from other methods), and how these might be used to improve model performance. Finally, we are interested in exploring how these types of explanations are actually interpreted by users, and whether providing them actually establishes trust in predictive systems.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Autism spectrum disorder (ASD) is a pervasive and highly heterogeneous neurodevelopmental condition. It is becoming increasingly apparent that systematic disentangling of this high heterogeneity is crucial for developing more precise diagnosis and targeted treatment strategies for ASD. Gender is a key source of heterogeneity in ASD. Indeed, one of the most consistent findings of epidemiological research is that ASD is diagnosed less frequently in females than in males, with a ratio of 1 to 4. 1 Additionally, converging evidence from behavioural studies suggests that females are less severely affected, especially in the restricted and repetitive behaviours (RRB) symptom domain, than males. 2 Yet, there is limited research examining the neurobiological differences between females and males with ASD. 3,4 Additionally, how gender differences in neurobiology relate to gender differences in the clinical symptomatology of the disorder is also not known. 3,4 Such knowledge is critical both for understanding the aetiology of this heterogeneous disorder and for determining neuroprotective mechanisms in females. 5 These knowledge gaps are partly because extant brain imaging studies have primarily focused on males and neglected females or used mixed samples involving a small number of females, 3,4 making it difficult to assess gender-related effects with adequate statistical power. Furthermore, extant studies have relied on conventional univariate approaches that are ill-equipped to capture robust neurobiological gender differences, necessitating the development of new computational approaches. 3 
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
The first aim of our study was to determine whether neurobiology, in particular functional brain organisation, differed between females and males with ASD. The number of studies examining gender differences in ASD at the brain level is minimal, 4 and the findings from these studies have been largely inconsistent: some studies have reported gender differences in functional brain organisation in ASD, whereas others have not found any. 3,4 Critically, the findings from these studies remain poorly replicated, likely because of the small numbers of participants, especially female participants. 3,4 To address this, we examine, to the best of our knowledge, one of the largest functional brain imaging data-sets to date of females and males with ASD obtained from multiple sites across the world-Autism Brain Imaging Data Exchange (ABIDE) 6 -along with the data we have collected, 7 as well as an independent Child Mind Institute-Health Brain Network (CMI-HBN) cohort, 8 using a novel explainable artificial intelligence 9 (XAI)-based framework.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
The field of XAI in recent years has been revolutionised by deep neural networks (DNNs). 10 DNNs, however, have been far less successful in classification/differentiation of groups using functional brain imaging data. 11 In fact, no study has employed DNNs to differentiate between females and males with ASD using functional brain imaging data. 3,4 This is because of the many challenges associated with applying DNNs to brain imaging data (see Supplementary Introduction available at https://doi.org/10.1192/ bjp.2022.13 for details). To address these challenges, we developed a novel spatiotemporal DNN (stDNN) model, which takes as its input functional magnetic resonance imaging (fMRI) time series data from brain regions of interest and models the underlying dynamic spatiotemporal characteristics of brain activity to differentiate between females with ASD and males with ASD ( Fig. 1 and Supplementary Fig. 1).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
A key idea of our approach is to discover latent spatiotemporal dynamics for classification from brain data without the need for explicit feature engineering. Another novel feature of our stDNN model is the use of a one-hot encoding scheme that addresses multisite heterogeneity in fMRI data that is common to data-sharing consortia such as ABIDE. The one-hot encoding scheme enabled the creation of a single stDNN model that handles heterogeneous data while learning robust representations for simultaneously classifying and identifying robust neurobiologically meaningful features that distinguish females and males with ASD. Yet another novel aspect of our stDNN is its use of a label-distribution-aware margin (LDAM) loss 12 during model training. DNNs fare poorly when the data-set suffers from heavy class imbalance, such as in the present case owing to well-known male bias of ASD. 1 To improve DNN performance in such scenarios, we designed a theoretically principled LDAM loss motivated by minimising a marginbased generalisation bound, which we have previously shown to outperform the conventional cross-entropy loss under class-imbalance conditions. 12 We applied our stDNN model with LDAM loss function to the multisite ABIDE 6 combined with Stanford 7 task-free fMRI data. We hypothesised that stDNN would be able to accurately distinguish between females and males with ASD in the ABIDE/Stanford cohort.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
To address growing concerns about reproducibility in neuroscience, we next applied our stDNN model with LDAM loss to CMI-HBN cohort task-free fMRI data. 8 It should be noted that the CMI-HBN cohort data was not used for training the stDNN model, and therefore it is a fully independent data-set for demonstrating the generalisability of the stDNN ASD gender classification model. This is a crucial step in which most approaches are widely known to fail. 11 We hypothesised that stDNN trained on ABIDE/Stanford cohort would be able to accurately distinguish between females and males with ASD in the fully independent CMI-HBN cohort.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
To determine the specificity of gender differences in functional brain organisation in ASD, we assessed whether the stDNN model trained to distinguish between females and males with ASD could also distinguish between neurotypical females and neurotypical males. We further hypothesised that the stDNN ASD gender classification model would not be able to accurately distinguish between neurotypical females and neurotypical males, reflecting unique patterns of gender differences in ASD. 2 The second aim of our study was to determine which aspects/ features of functional brain organisation differed between females and males with ASD. Conventional DNN approaches are black box models, which provide no insight into which brain features are important for classification, nor whether the features are neurobiologically interpretable in the context of previous research of gender differences in ASD. 11 We address this gap by using an integrated gradients method 13 for identifying neurobiologically meaningful features that distinguish between females and males with ASD. This method ranks brain features that distinguish between females and males with ASD. We hypothesised that integrated gradients would reveal functional organisation patterns that are different between females and males with ASD in multiple brain areas, particularly those belonging to the motor and language networks. 2 We further predicted that the patterns of ASD gender differences in functional brain organisation would differ from the normative/ typical gender difference patterns. 2 The third aim of our study was to examine the relationship between the functional brain organisation patterns that differ in females and males with ASD and their symptom severity. To our knowledge, no previous study has examined functional brain features that robustly predict clinical symptoms in females and males with ASD separately without using feature engineering. We hypothesised that the brains of females and males with ASD would be functionally organised in ways that contribute differently to their clinical symptoms.  Fig. 1 Schematic overview of multicomponent explainable artificial intelligence (XAI) framework for discovering neurobiological patterns/ fingerprints that distinguish between females and males with autism spectrum disorder (ASD) and predict the severity of clinical symptoms. Key steps include: Steps 1, 2: data extraction;
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Step 3, 4: classification; Steps 5, 6: feature identification. i.e. predictive feature weights ('fingerprints') across brain regions; and
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Step 7: prediction of clinical symptom severity.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
ADI-R, autism diagnostic interview-revised; Avg, average; F, filter; fMRI, functional magnetic resonance imaging; Nc, number of brain regions; Nt, number of time points; ReLU, rectified linear unit; S, stride; stDNN, spatiotemporal deep neural network. legal guardian. The study protocol was approved by the Stanford University Institutional Review Board. Specifically, the ABIDE data was combined with the Stanford data to form the ABIDE/ Stanford cohort, which served as the primary cohort.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
An independent cohort of participants from CMI-HBN 8 was used to demonstrate the robustness of our findings. (Supplementary  Table 2 
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We developed an innovative stDNN model to extract informative functional brain dynamics features that accurately distinguish between females and males with ASD (see Supplementary Methods for details). Briefly, our stDNN model consists of two 1D-convolutional block layers, a 'temporal averaging' operation, and then a sigmoid output layer ( Supplementary Fig. 1). Pre-processed regional fMRI time series from 246 brain regions defined in the Brainnetome atlas were given as input to the first 1D-convolutional block layer. To account for site-related heterogeneity, site information encoded with a one-hot encoding scheme was given as an input to the final layer. stDNN classified participants in the two groups by minimising a LDAM loss function.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
To prevent bias and account for low variance, we conducted a fivefold cross-validation to evaluate the performance (accuracy, precision, recall, and the harmonic mean of precision and recall (F1)) of our stDNN model in classifying ASD females versus ASD males ( Fig. 2; see Supplementary Methods for details).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Neurotypical gender classification analysis of ABIDE/ Stanford cohort data using fivefold ABIDE/Stanford cohort ASD gender classification models
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
To examine the specificity of our stDNN ASD gender classification model, we investigated whether the stDNN model trained to distinguish between females with ASD and males with ASD can  Fig. 2 Fivefold cross-validation procedure for testing and validation of females with autism spectrum disorder (ASD) versus males with ASD classification using the ABIDE/Stanford cohort. The five models are then used for independently testing females with ASD versus males with ASD classification in the Child Mind Institute-Health Brain Network (CMI-HBN) cohort.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Note that these models are not trained on the CMI-HBN cohort. ADI-R, autism diagnostic interview-revised; Avg, average; CV, cross-validation. distinguish between neurotypical females and neurotypical males in the ABIDE/Stanford cohort (see Supplementary Methods for details).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We conducted a fivefold cross-validation to evaluate the performance (accuracy, precision, recall, F1) of our de novo trained stDNN model in classifying neurotypical females versus neurotypical males (see Supplementary Methods for details).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We used an integrated gradients-based feature attribution approach 13 (see Supplementary Methods for details) to identify brain features that discriminated between females with ASD and males with ASD.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We investigated the relationship between stDNN-identified neurobiological features (that distinguished between females and males with ASD) with the severity of clinical symptoms in females with ASD and males with ASD separately. Spearman correlations between the autism diagnostic interview-revised (ADI-R) 14 domain scores and the brain features derived from each of the five stDNN models were computed (see Supplementary Methods for details).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We performed several control analyses to demonstrate that our findings are robust to head motion-related confounds (see Supplementary Methods for details).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We first trained our stDNN on the multisite ABIDE cohort data combined with data we acquired at Stanford (ASD cohort n = 678) (Supplementary Table 1). To assess the performance of our stDNN model, we used a fivefold cross-validation procedure in which 80% of the sample was used for training and the other 20% of the sample was used for validation (Fig. 2) Table 3). Additional analyses confirmed that the observed results were robust to potential confounds such as head motion (see Supplementary Results for details). These results demonstrate that stDNN can accurately distinguish females with ASD from males with ASD in a multisite cohort, and, furthermore, does so in a robust and consistent manner across cross-validation folds.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We then evaluated the performance of our stDNN model on an independent cohort of females with ASD and males with ASD obtained from CMI-HBN (ASD cohort n = 95) (Supplementary Table 2). Importantly, the stDNN was not trained on the CMI-HBN data. We evaluated five models corresponding to each of the folds in the cross-validation as described above (Fig. 2). stDNN  Table 4). These results demonstrate that stDNN can accurately distinguish females with ASD from males with ASD in an independent cohort without additional training.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
To examine the specificity of our stDNN ASD gender classification model, we investigated whether the stDNN model trained to distinguish between females with ASD and males with ASD can distinguish between neurotypical females and neurotypical males in the ABIDE/ Stanford cohort (neurotypical cohort n = 976) (Supplementary Table 1 Table 6).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
These results point to a unique pattern of gender differences in ASD.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We then used an integrated gradients procedure 13 to compute the feature attributes underlying the ASD female class label in the ABIDE/Stanford cohort. This analysis yields a measure of feature strength associated with females with ASD versus males with ASD classification in each brain region and at each time point. The integrated gradients procedure was applied to the stDNN model that is trained to distinguish between females with ASD and males with ASD using ASD data from the ABIDE/Stanford cohort. This procedure also identifies an individual fingerprint of predictive features in each participant (Fig. 3, see Supplementary Results for details).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
To identify brain areas that contributed the most to classification, we computed the median of feature scores across the five folds and thresholded themtop 5% of featuresbased on the distribution of feature scores across all time points and regions. This resulted in the identification of a distributed set of brain areas including the primary motor cortex, supplementary motor area, parietal and lateral occipital cortex, and middle and superior temporal gyri as brain areas that contribute most significantly to predicting the ASD female class label ( Fig. 4 and Supplementary Table 7). These brain areas were distinct from those that contributed most significantly to predicting the neurotypical female class label (see Supplementary Results for details).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Additional analyses confirmed that the observed results were robust to potential confounds such as head motion (see Supplementary Results for details). These results demonstrate that stDNN together with integrated gradients procedures automatically identifies discriminating features without the need for ad hoc feature engineering procedures.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We then used the same procedures as described in the previous section to determine predictive feature attributes in each female Gender differences in functional brain organisation in autism participant with ASD in the CMI-HBN cohort. This analysis revealed individualised brain 'fingerprints' (Fig. 3, see Supplementary Results for details), and identified the primary motor cortex, supplementary motor area, parietal and lateral occipital cortex, and middle and superior temporal gyri as the brain areas that contributed most significantly to predicting the ASD female class label (Fig. 4 and Supplementary Table 8).
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
These results demonstrate that stDNN together with integrated gradients procedures automatically identifies similar discriminating features as in the ABIDE/Stanford cohort, again without the need for ad hoc feature engineering procedures.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
We investigated whether stDNN-identified brain features could predict the severity of clinical symptoms in females and males with ASD. The primary motor cortex was the only brain region whose features predicted ADI-R RRB scores (P < 0.01; false discovery rate-corrected) in females with ASD. No such relationship was observed in males with ASD or for the social and communication domains in either females or males with ASD, demonstrating the specificity of findings related to RRB phenotypic features in females with ASD.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
By using one of the largest functional brain imaging cohorts of females and males with ASD and leveraging exciting recent advances in XAI, we examined neurobiological gender differences in ASD. To our knowledge, this is the first use of a XAI-based approach for uncovering robust neurobiological gender differences in ASD. Our XAI-based approach is also a significant advance over previous approaches to find gender differences in ASD using functional brain imaging data (see Supplementary Discussion for details). Our novel XAI-based stDNN model, which uses spatiotemporal convolution on fMRI data to distinguish between groups, achieved consistently high classification accuracy in distinguishing between females and males with ASD. Notably, the stDNN model trained to distinguish between females and males with ASD could not distinguish between neurotypical females and males, suggesting that there are gender differences in the functional brain organisation in ASD and these gender differences are different from normative gender differences.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Brain features associated with motor, language and visuospatial attentional systems reliably distinguished females with ASD from males with ASD. Crucially, these results were observed in the  large multisite ABIDE/Stanford cohort and replicated in a fully independent CMI-HBN cohort. Furthermore, brain features associated with the primary motor cortex node of the motor network emerged as a robust predictor of the severity of RRBs in females with ASD but not in males with ASD. Taken together, these results, elaborated below, provide novel and robust insights into the neurobiology of gender differences in ASD and their relation to the core clinical symptoms of the disorder.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
The first key finding of our study is that females with ASD have a functional brain organisation that differs significantly from males with ASD. Gender differences in functional brain organisation in ASD reported to date have been mixed and, to the best of our knowledge, no studies have attempted model-based validation and replication of findings. stDNN allowed us to perform cross-validation analysis, a powerful approach for validating research findings, and its use for demonstrating generalisation and reproducibility is now increasingly advocated in psychiatry. 15 Notably, stDNN also allowed us to develop a model based on one data-set (ABIDE/Stanford) and test it on another 'independent' data-set (CMI-HBN) without further training. This is a crucial step in which most approaches in other domains are widely known to fail. 15 Our stDNN model learned functional brain organisation patterns that could distinguish between females and males with ASD with a high cross-validation classification accuracy in the multisite ABIDE/Stanford cohort, despite the considerable heterogeneity in data acquisition protocols and a wide range in age (mean age = 13.2 years, s.d. = 5.9) and symptom profiles. Notably, stDNN achieved a high classification accuracy in an independent CMI-HBN cohort (mean age = 11.8 years, s.d. = 3.9) that was not used in the training of stDNN. These results further demonstrate robust gender differences in functional brain organisation in ASD.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
stDNN also enabled us to seamlessly determine whether functional brain patterns that distinguished between females and males with ASD could also classify by gender in data from neurotypical peers. We found that the stDNN model trained to distinguish between females and males with ASD could not distinguish between neurotypical females and males. These results point to unique neurofunctional patterns of gender differences in ASD.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
The second main finding of our study is the identification of functional brain features that distinguish between females and males with ASD. Our stDNN-based integrated gradients analysis identified the primary motor cortex and supplementary motor area, which anchor the motor network, as brain areas whose dynamical functional properties most clearly distinguished between females and males with ASD in both cohorts. Aberrancies in the extended motor network have been consistently reported in ASD, when compared with neurotypical individuals. 16 An open unaddressed question is whether these same brain areas also show gender-specific differentiation, which our findings help resolve in a robust manner across multiple cohorts. This question is important because aberrancies in key nodes of the motor network and their dynamic functional interactions have been linked to fine and gross motor deficits in ASD, 16 and there is increasing evidence for gender differences in motor deficits in ASD. 17 Previous studies have reported ASD-related gender differences in grey matter morphometry of the motor network, albeit using small sample sizes. 2 Our findings, using a large sample of females and males with ASD, extend these results by providing novel and robust evidence that the functional brain organisation of the motor system, which has been consistently shown to differ between ASD and their neurotypical peers, is also different between females and males with ASD. with integrated gradients identified brain features that distinguish females with ASD from males with ASD. The algorithm automatically identified distinguishing features in the primary motor cortex and the supplementary motor area, which anchor the motor network, middle and superior temporal gyri, which anchor the language network, as well as the visuospatial attentional system (see Supplementary Table 7 for a detailed listing of brain areas and predictive feature weights). (b) Visualisation of (unthresholded) feature weights across the whole brain in the ABIDE/Stanford cohort. (c) Feature attribution map showing the top 5% features showing replication of predictive motor network, language network, and visuospatial attention features in the Child Mind Institute-Health Brain Network (CMI-HBN) cohort (see Supplementary Table 8 for a detailed listing of brain areas and predictive feature weights). (d) Visualisation of (unthresholded) feature weights across the whole brain in the CMI-HBN cohort.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
In both cohorts, our analysis also identified bilateral middle and superior temporal gyri, which anchor the language system, as brain areas whose features clearly distinguished between females and males with ASD. Impairments in temporal cortex areas associated with language processing are a prominent feature of ASD. 18,19 Language systems anchored in the middle and superior temporal gyri facilitate speech processing and semantic comprehension, processes known to be impaired in ASD. 20 Surprisingly, our analysis also identified the dorsal parietal cortex as a brain area whose features distinguished between females and males with ASD in both cohorts. Specific loci included the superior parietal lobule and the adjoining posterior human intraparietal area 3 (hIP3) subdivision of the intraparietal sulcus, which play a crucial role in visuospatial attention. 21,22 Moreover, these parietal areas, encompassing the dorsal visual stream pathway, have strong connections with the lateral occipital gyrus 21 whose features also distinguished between females and males with ASD. Together, these regions constitute key elements of the dorsal and ventral visual pathways involved in attending to the location of objects in space. 23 This finding of gender-related differences is noteworthy because of conflicting evidence in the extant literature about co-occurring parietal and lateral occipital cortex impairments in ASD. 18,19 Taken together, these results identify gender differences in motor and language systems that are known to be impaired in the disorder as well as dorsal parietal and lateral occipital cortex regions that have not been reported previously in the literature as being affected in ASD. Interestingly, although default mode network and salience network aberrancies have been consistently reported in ASD, 24,25 gender differences were not reliably observed in these regions, suggesting that aberrancies in the default mode network and salience network may be common to females and males with ASD. More broadly, our findings suggest that brain characteristics of gender differences in ASD overlap only partly with overall brain aberrancies in ASD, emphasising the convergent and distinct underlying mechanisms of gender differences in the disorder.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
These observations parallel findings from neuroanatomical studies that suggest that the overall ASD neuroanatomy in females and males does not simply differ quantitatively in the same brain regions/circuitries but also differs qualitatively. 26 The extent to which abnormalities in the brain systems identified in the present study differentially have an impact on the acquisition of motor, visuospatial and social communication skills in females and males with ASD remains to be investigated.
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
A third important finding of our study is that we found that the primary motor cortex node of the motor network was the only brain area whose dynamic functional circuitry predicted scores on the RRB domain of the ADI-R. They were not related to the scores on the social and communication scores of the ADI-R, indicating domain-specific effects associated with RRBsa core clinical phenotype of ASD that has been most consistently reported to differ between genders. 2 Surprisingly, these relationships were observed in females, but not in males, with ASD. The neurofunctional mechanisms underlying the heterogeneity of RRB symptoms in males with ASD remain to be investigated. A potential avenue for investigation might be to examine the relationship between RRBs in males with ASD and the functional organisation of other components of the motor network, including the basal ganglia and cerebellum, which we recently found to be predictive of repetitive motor behaviours in a predominantly male ASD sample. 7
Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism
Our findings reveal that the brains of females and males with ASD are functionally organised differently, contributing to their clinical symptoms in distinct ways. Our discovery of robust neurobiological gender differences in ASD psychopathology has the potential to transform our understanding of the diverse aetiologies of the dis-
Axiomatic Attribution for Deep Networks
We study the problem of attributing the prediction of a deep network to its input features.
Axiomatic Attribution for Deep Networks
Definition 1. Formally, suppose we have a function F : R n → [0, 1] that represents a deep network, and an input x = (x 1 , . . . , x n ) ∈ R n . An attribution of the prediction at input x relative to a baseline input x is a vector A F (x, x ) = (a 1 , . . . , a n ) ∈ R n where a i is the contribution of x i to the prediction F (x).
Axiomatic Attribution for Deep Networks
For instance, in an object recognition network, an attribution method could tell us which pixels of the image were responsible for a certain label being picked (see Figure 2). The attribution problem was previously studied by various papers (Baehrens et al., 2010;Simonyan et al., 2013;Shrikumar et al., 2016;Binder et al., 2016;Springenberg et al., 2014).
Axiomatic Attribution for Deep Networks
The intention of these works is to understand the inputoutput behavior of the deep network, which gives us the ability to improve it. Such understandability is critical to all computer programs, including machine learning models. There are also other applications of attribution. They could be used within a product driven by machine learning to provide a rationale for the recommendation. For instance, a deep network that predicts a condition based on imaging could help inform the doctor of the part of the image that resulted in the recommendation. This could help the doctor understand the strengths and weaknesses of a model and compensate for it. We give such an example in Section 5.2. Attributions could also be used by developers in an exploratory sense. For instance, we could use a deep network to extract insights that could be then used in a rulebased system. In Section 5.3, we give such an example. A significant challenge in designing an attribution technique is that they are hard to evaluate empirically. As we discuss in Section 4, it is hard to tease apart errors that stem from the misbehavior of the model versus the misbehavior of the attribution method. To compensate for this shortcoming, we take an axiomatic approach. In Section 2 we identify two axioms that every attribution method must satisfy. Unfortunately most previous methods do not satisfy one of these two axioms. In Section 3, we use the axioms to identify a new method, called integrated gradients.
Axiomatic Attribution for Deep Networks
Unlike previously proposed methods, integrated gradients do not need any instrumentation of the network, and can be computed easily using a few calls to the gradient opera-arXiv:1703.01365v1 [cs.LG] 4 Mar 2017 tion, allowing even novice practitioners to easily apply the technique.
Axiomatic Attribution for Deep Networks
In Section 5, we demonstrate the ease of applicability over several deep networks, including two images networks, two text processing networks, and a chemistry network. These applications demonstrate the use of our technique in either improving our understanding of the network, performing debugging, performing rule extraction, or aiding an end user in understanding the network's prediction.
Axiomatic Attribution for Deep Networks
Remark 1. Let us briefly examine the need for the baseline in the definition of the attribution problem. A common way for humans to perform attribution relies on counterfactual intuition. When we assign blame to a certain cause we implicitly consider the absence of the cause as a baseline for comparing outcomes. In a deep network, we model the absence using a single baseline input. For most deep networks, a natural baseline exists in the input space where the prediction is neutral. For instance, in object recognition networks, it is the black image. The need for a baseline has also been pointed out by prior work on attribution (Shrikumar et al., 2016;Binder et al., 2016).
Axiomatic Attribution for Deep Networks
We now discuss two axioms (desirable characteristics) for attribution methods. We find that other feature attribution methods in literature break at least one of the two axioms. These methods include DeepLift (Shrikumar et al., 2016), Layer-wise relevance propagation (LRP) (Binder et al., 2016), Deconvolutional networks (Zeiler & Fergus, 2014), and Guided back-propagation (Springenberg et al., 2014). As we will see in Section 3, these axioms will also guide the design of our method.
Axiomatic Attribution for Deep Networks
Gradients. For linear models, ML practitioners regularly inspect coefficients of the model in order to debug it. Gradients (of the output with respect to the input) are a natural analog of these coefficients for a deep network, and therefore is a reasonable starting point for an attribution method (Baehrens et al., 2010;Simonyan et al., 2013); see the third column of Figure 2 for examples. The problem with gradients is that they break sensitivity, a property that all attribution methods should satisfy.
Axiomatic Attribution for Deep Networks
An attribution method satisfies Sensitivity(a) if for every input and baseline that differ in one feature but have different predictions then the differing feature should be given a non-zero attribution. (Later in the paper, we will have a part (b) to this definition.)
Axiomatic Attribution for Deep Networks
Gradients violate Sensitivity(a): For a concrete example, consider a one variable, one ReLU network, f (x) = 1 − ReLU(1−x). Suppose the baseline is x = 0 and the input is x = 2. The function changes from 0 to 1, but because f becomes flat at x = 1, the gradient method gives attribution of 0 to x. Intuitively, gradients break Sensitivity because the prediction function may flatten at the input and thus have zero gradient despite the function value at the input being different from that at the baseline. This phenomenon has been reported in previous work (Shrikumar et al., 2016).
Axiomatic Attribution for Deep Networks
Practically, the lack of sensitivity causes gradients to focus on irrelevant features (see the "fireboat" example in Figure 2).
Axiomatic Attribution for Deep Networks
Other back-propagation based approaches. A second set of approaches involve back-propagating the final prediction score through each layer of the network down to the individual features. These include DeepLift (Shrikumar et al., 2016), Layer-wise relevance propagation (LRP) (Binder et al., 2016), Deconvolutional networks (De-ConvNets) (Zeiler & Fergus, 2014), and Guided backpropagation (Springenberg et al., 2014). These methods differ in the specific backpropagation logic for various activation functions (e.g., ReLU, MaxPool, etc.).
Axiomatic Attribution for Deep Networks
Deconvolution networks (DeCon-vNets) (Zeiler & Fergus, 2014), and Guided backpropagation (Springenberg et al., 2014) violate Sensitivity(a). This is because these methods back-propogate through a ReLU node only if the ReLU is turned on at the input. This makes the method similar to gradients, in that, the attribution is zero for features with zero gradient at the input despite a non-zero gradient at the baseline. We defer the specific counterexamples to the Appendix B.
Axiomatic Attribution for Deep Networks
Methods like DeepLift and LRP tackle the Sensitivity issue by employing a baseline, and in some sense try to compute "discrete gradients" instead of (instantaeneous) gradients at the input. (The two methods differ in the specifics of how they compute the discrete gradient). But the idea is that a large, discrete step will avoid flat regions, avoiding a breakage of sensitivity. Unfortunately, these methods violate a different requirement that attribution methods should satisfy.
Axiomatic Attribution for Deep Networks
Two networks are functionally equivalent if their outputs are equal for all inputs, despite having very different implementations. Attribution methods should satisfy Implementation Invariance, i.e., the attributions are always identical for two functionally equivalent networks. To motivate this, notice that attribution can be colloquially defined as assigning the blame (or credit) for the output to the input features. Such a definition does not refer to implementation details.
Axiomatic Attribution for Deep Networks
We now discuss intuition for why DeepLift and LRP break Implementation Invariance; a concrete example is provided in Appendix B. First, notice that gradients are invariant to implementation. In fact, the chain-rule for gradients ∂f ∂g = ∂f ∂h · ∂h ∂g is essentially about implementation invariance. To see this, think of g and f as the input and output of a system, and h being some implementation detail of the system. The gradient of output f to input g can be computed either directly by ∂f ∂g , ignoring the intermediate function h (implementation detail), or by invoking the chain rule via h. This is exactly how backpropagation works.
Axiomatic Attribution for Deep Networks
Methods like LRP and DeepLift replace gradients with discrete gradients and still use a modified form of backpropagation to compose discrete gradients into attributions. 1Unfortunately, the chain rule does not hold for discrete gradients in general. x0) , and therefore these methods fail to satisfy implementation invariance.
Axiomatic Attribution for Deep Networks
If an attribution method fails to satisfy Implementation Invariance, the attributions are potentially sensitive to unimportant aspects of the models. For instance, in the example in the appendix, the network architecture has more degrees of freedom than needed for representing the function, and as a result there are two set of values for the network parameters that lead to the same function. The training procedure can converge at either set of values depending on the initializtion or for other reasons, but the underlying network function would remain the same. It is undesirable that attributions differ for such reasons.
Axiomatic Attribution for Deep Networks
We are now ready to describe our technique. Intuitively, our technique combines the Implementation Invariance of Gradients along with the Sensitivity of techniques like LRP or DeepLift.
Axiomatic Attribution for Deep Networks
Formally, suppose we have a function F : R n → [0, 1] that represents a deep network. Specifically, let x ∈ R n be the input at hand, and x ∈ R n be the baseline input. For image networks, the baseline could be the black image, while for text models it could be the zero embedding vector.
Axiomatic Attribution for Deep Networks
We consider the straightline path (in R n ) from the baseline x to the input x, and compute the gradients at all points along the path. Integrated gradients are obtained by cumulating these gradients. Specifically, integrated gradients are defined as the path intergral of the gradients along the straightline path from the baseline x to the input x.
Axiomatic Attribution for Deep Networks
The integrated gradient along the i th dimension for an input x and baseline x is defined as follows.
Axiomatic Attribution for Deep Networks
where ∂F (x) ∂xi is the gradient of F along the i th dimension at x.
Axiomatic Attribution for Deep Networks
Axiom: Completeness. Integrated gradients satisfy an axiom called completeness that the attributions add up to the difference between the output of F at the input x and the baseline x . This axiom is identified as being desirable by Deeplift and LRP. It is a sanity check that the attribution method is somewhat comprehensive in its accounting, a property that is clearly desirable if the networks score is used in a numeric sense, and not just to pick the top label, for e.g., a model estimating insurance premiums from credit features of individuals. This is formalized by the proposition below, which is an instantiation of the fundamental theorem of calculus for path integrals.
Axiomatic Attribution for Deep Networks
For most deep networks, it is possible to choose a baseline such that the prediction at the baseline is near zero (F (x ) ≈ 0). (For image models, the black image baseline indeed satisfies this property.) In such cases, there is an intepretation of the resulting attributions that ignores the baseline and amounts to distributing the output to the individual input features.
Axiomatic Attribution for Deep Networks
Remark 2. Integrated Gradients satisfies Sensivity(a) because Completeness implies Sensivity(a) and is thus a strengthening of the Sensitivity(a) axiom. This is because Sensitivity(a) refers to a case where the baseline and the input differ only in one variable, for which Completeness asserts that the difference in the two output values is equal to the attribution to this variable. Attributions generated by integrated gradients satisfy Implementation Invariance since they are based only on the gradients of the function represented by the network.
Axiomatic Attribution for Deep Networks
Computing integrated gradients. The integrated gradients can be efficiently approximated by Riemann sum, wherein, we simply sum the gradients at points occurring at sufficiently small intervals along the straightline path from the baseline x to the input x.
Axiomatic Attribution for Deep Networks
Here m is the number of steps in the Riemman approximation of the integral. Notice that the approximation simply involves computing the gradient in a for loop which should be straightforward and efficient in most deep learning frameworks. For instance, in TensorFlow, it essentially amounts to calling tf.gradients in a loop over the set of inputs (i.e., x + k m × (x − x ) for k = 1, . . . , m), which could also be batched. In practice, we find that m = 50 calls are more than enough to approximate the integral to within 5%. Going forward, we abuse the term "integrated gradients" to refer to the approximation described above.
Axiomatic Attribution for Deep Networks
Prior literature has relied on empirically evaluating the attribution technique. For instance, in the context of an object recognition task, (Samek et al., 2015) suggests that we select the top k pixels by attribution and randomly vary their intensities and then measure the drop in score. If the attribution method is good, then the drop in score should be large. However, the images resulting from pixel perturbation could be unnatural, and it could be that the scores drop simply because the network has never seen anything like it in training. (This is probably less of a concern with linear or logistic models where the simplicity of the model ensures that ablating a feature does not cause strange interactions.)
Axiomatic Attribution for Deep Networks
A different evaluation technique considers images with human-drawn bounding boxes around objects, and computes the percentage of pixel attribution inside the bounding box. While for most objects, one would expect the pixels located on the object to be most important for the prediction, in some cases the context in which the object occurs may also contribute to the prediction. The cabbage butterfly image from Figure 2 is a good example of this where the pixels on the leaf are also surfaced by the integrated gradients.
Axiomatic Attribution for Deep Networks
Roughly, we found that every empirical evaluation technique we could think of could not differentiate between artifacts that stem from perturbing the data, a misbehaving model, and a misbehaving attribution method. This was why we turned to an axiomatic approach in designing a good attribution method (Section 2). While our method satisfies Sensitivity and Implementation Invariance, it certainly isn't the unique method to do so.
Axiomatic Attribution for Deep Networks
We now justify the selection of the integrated gradients method in two steps. First, we identify a class of methods called Path methods that generalize integrated gradients. We discuss that path methods are the only methods to satisfy certain desirable axioms. Second, we argue why integrated gradients is somehow canonical among the different path methods. r 1 ,r 2 s 1 ,s 2 P 1 P 2 P 3 Figure 1: Three paths between an a baseline (r 1 , r 2 ) and an input (s 1 , s 2 ). Each path corresponds to a different attribution method. The path P 2 corresponds to the path used by Integrated Gradients.
Axiomatic Attribution for Deep Networks
Integrated gradients aggregate the gradients along the inputs that fall on the straightline between the baseline and the input. Clearly, there are many other (nonstraightline) paths that monotonically interpolate between the two points, and each such path will yield a different attribution method. For instance, consider the simple case when the input is two dimensional. Figure 1 has examples of three paths, each of which corresponds to a different attribution method.
Axiomatic Attribution for Deep Networks
Formally, let γ = (γ 1 , . . . , γ n ) : [0, 1] → R n be a smooth function specifying a path in R n from the baseline x to the input x, i.e., γ(0) = x and γ(1) = x.
Axiomatic Attribution for Deep Networks
Given a path function γ, path integrated gradients are obtained by integrating the gradients along the path γ(α) for α ∈ [0, 1]. Formally, path integrated gradients along the i th dimension for an input x is defined as follows.
Axiomatic Attribution for Deep Networks
∂xi is the gradient of F along the i th dimension at x.
Axiomatic Attribution for Deep Networks
Attribution methods based on path integrated gradients are collectively known as path methods. Notice that integrated gradients is a path method for the straightline path specified
Axiomatic Attribution for Deep Networks
Remark 3. All path methods satisfy Implementation Invariance. This follows from the fact that they are defined using the underlying gradients, which do not depend on the implementation. They also satisfy Sensitvity(a) because all path methods satisfy Completeness (the proof is similar to that of Proposition 1) which implies Sensitivity(a) (see Remark 2).
Axiomatic Attribution for Deep Networks
More interestingly, path methods are the only methods that satisfy certain desirable axioms. (For formal defini-tions of the axioms and proof of Proposition 2, see Friedman (Friedman, 2004).)
Axiomatic Attribution for Deep Networks
Axiom: Sensitivity(b). (called Dummy in (Friedman, 2004)) If the function implemented by the deep network does not depend (mathematically) on some variable, then the attribution to that variable is always zero. This is a natural complement to the definition of Sensitivity(a) from Section 2. This definition captures desired insensitivity of the attributions.
Axiomatic Attribution for Deep Networks
Axiom: Linearity. Suppose that we linearly composed two deep networks modeled by the functions f 1 and f 2 to form a third network that models the function a×f 1 +b×f 2 , i.e., a linear combination of the two networks. Then we'd like the attributions for a × f 1 + b × f 2 to be the weighted sum of the attributions for f 1 and f 2 with weights a and b respectively. Intuitively, we would like the attributions to preserve any linearity within the network.
Axiomatic Attribution for Deep Networks
Proposition 2. (Theorem 1 (Friedman, 2004)) Path methods are the only attribution methods that always satisfy Implementation Invariance, Sensitivity(b), Linearity, and Completeness.
Axiomatic Attribution for Deep Networks
Remark 4. We note that these path integrated gradients have been used within the cost-sharing literature in economics where the function models the cost of a project as a function of the demands of various participants, and the attributions correspond to cost-shares. Integrated gradients correspond to a cost-sharing method called Aumann-Shapley (Aumann & Shapley, 1974). Proposition 2 holds for our attribution problem because mathematically the cost-sharing problem corresponds to the attribution problem with the benchmark fixed at the zero vector. (Implementation Invariance is implicit in the cost-sharing literature as the cost functions are considered directly in their mathematical form.)
Axiomatic Attribution for Deep Networks
In this section, we formalize why the straightline path chosen by integrated gradients is canonical. First, observe that it is the simplest path that one can define mathematically. Second, a natural property for attribution methods is to preserve symmetry, in the following sense. E.g., consider the logistic model Sigmoid(x 1 + x 2 + . . . ).
Axiomatic Attribution for Deep Networks
x 1 and x 2 are symmetric variables for this model. For an input where x 1 = x 2 = 1 (say) and baseline where x 1 = x 2 = 0 (say), a symmetry preserving method must offer identical attributions to x 1 and x 2 .
Axiomatic Attribution for Deep Networks
It seems natural to ask for symmetry-preserving attribution methods because if two variables play the exact same role in the network (i.e., they are symmetric and have the same values in the baseline and the input) then they ought to receive the same attrbiution.
Axiomatic Attribution for Deep Networks
Theorem 1. Integrated gradients is the unique path method that is symmetry-preserving.
Axiomatic Attribution for Deep Networks
The integrated gradients technique is applicable to a variety of deep networks. As a proof of concept, we apply the technique to two image models, two natural language models, and a chemistry model.
Axiomatic Attribution for Deep Networks
We study feature attribution in an object recognition network built using the GoogleNet architecture (Szegedy et al., 2014) and trained over the ImageNet object recognition dataset (Russakovsky et al., 2015). We use the integrated gradients method to study pixel importance in predictions made by this network. The gradients are computed for the output of the highest-scoring class with respect to pixel of the input image. The baseline input is the black image, i.e., all pixel intensities are zero. Integrated gradients essentially aggregate the gradients of images obtained by interpolating between the original image and this black image.
Axiomatic Attribution for Deep Networks
Integrated gradients can be visualized by aggregating them along the color channel and scaling the pixels in the actual image by them. Figure 2 shows visualizations for a bunch of images 2 . For comparison, it also presents the corresponding visualization obtained from the gradients at the actual image. Notice that integrated gradients are better at reflecting distinctive features of the image compared to just gradients at the input.
Axiomatic Attribution for Deep Networks
To further quantify the effectiveness of integrated gradients over gradients, we perform two analyses.
Axiomatic Attribution for Deep Networks
1. Pixel ablations: Based on a method proposed by (Samek et al., 2015), we ablate (i.e, zero the pixel intensity) the top 5000 pixels (10% of the image) by attribution, and compute the score drop for the highest scoring object class. We find that ablating the top pix-els identified by integrated gradients leads to a larger score drop that those identified by gradients at the image for 132 out of 150 randomly chosen images.
Axiomatic Attribution for Deep Networks
We consider images from the 2012 Im-ageNet object localization dataset with human-drawn bounding boxes around objects, and compute the percentage of pixel attribution inside the bounding box. We run our evaluation over 100 randomly chosen images and find that on 82 images the integrated gradients technique leads to a higher fraction of the pixel attribution inside the box than gradients at the actual image. The average difference in the percentage pixel attribution inside the box for the two techniques is 8.4%.
Axiomatic Attribution for Deep Networks
Comparing integrated gradients with gradients at the image. Left-to-right: original input image, label and softmax score for the highest scoring class, visualization of integrated gradients, visualization of gradients at the image. Notice that the visualizations obtained from integrated gradients are better at reflecting distinctive features of the image.
Axiomatic Attribution for Deep Networks
Diabetic retinopathy (DR) is a complication of the diabetes that affects the eyes. Recently, a deep network (V et al., 2016) has been proposed to predict the severity grade for DR in retinal fundus images. The model has good predictive accuracy on various validation datasets.
Axiomatic Attribution for Deep Networks
We use integrated gradients to study feature importance for this network; like in the object recognition case, the baseline is the black image. Feature importance explanations are important for this network as surfacing them to retina specialists helps to encourage clinical adoption. Retina specialists may use it to build trust in the network's predictions, decide the grade for borderline cases, and obtain insights for further testing and screening. Figure 3 shows a visualization of integrated gradients for a retinal fundus image. The visualization is obtained by scaling the integrated gradients and overlaying them along the green channel on the actual image in gray scale. Notice that integrated gradients seem to be localized to a few pixels that seem to be lesions in the retina.
Axiomatic Attribution for Deep Networks
The evaluated integrated gradients as part of wider study with retina specilists and found that integrated gradients help the specialist confirm the predicted DR grade on 28 out of 33 images from the EyePACs-1 dataset (V et al., 2016) chosen for diversity. We are currently scaling this study. Figure 3: Attribution for Diabetic Retinopathy grade prediction from a retinal fundus image The original image is show on the left, and the attributions (overlayed on the original image in gray scaee) is shown on the right. On the original image we annotate lesions visible to a human, and confirm that the attributions indeed point to them.
Axiomatic Attribution for Deep Networks
Automatically answering natural language questions (over semi-structured data) is an important problem in artificial intelligence (AI). A common approach is to semantically parse the question to its logical form (Liang, 2016;Pasupat & Liang, 2015) using a set of human-authored grammar rules. An alternative approach is to machine learn an endto-end model provided there is enough training data.
Axiomatic Attribution for Deep Networks
Rule based systems have the advantage of being more precise at language understanding (Liang, 2016), more main-tainable and easy to debug. The downside is that they have poorer recall. This is because there are often multiple phrasing of the same question in natural language, e.g., "how many people work at Walmart?" and "what is the number of employees at Walmart?" It is hard for humanauthored rules to be comprehensive on all possible phrasings. This is where machine learnt models are better.
Axiomatic Attribution for Deep Networks
An interesting question then is whether one could use peek inside machine learnt models to derive new rules for improving recall of the overall system. We explore this direction for a sub-problem of semantic parsing, called question classification, using the method integrated gradients.
Axiomatic Attribution for Deep Networks
The goal of question classification is to identify the intent of the question as to the type of answer it is seeking. For instance, is the quesiton seeking a yes/no answer, or is it seeking a date? Rules of solving this problem look for trigger phrases in the question, for e.g., a "when" in the beginning indicates a date seeking question. We train a model for question classification using the the text categorization architecture proposed by (Kim, 2014) over the WikiTableQuestions dataseet (Pasupat & Liang, 2015). We use integrated gradients to attribute predictions down to the question terms in order to identify new trigger phrases for answer type. In a sense, we are using attributions as a dataexploration tool. The baseline input is the all zero embedding vector. Unlike in the case of image models, the input on the straightline path from the baseline and the original input no longer have a natural interpretation. Our method works for the reasons discussed in Section 2 and 4. Figure 4 lists a few questions with constituent terms highlighted based on their attribution. Notice that the attributions largely agree with commonly used rules, for e.g., "how many" indicates a numeric seeking question. In addition, attributions also help identify novel question classification rules. For instance, questions containing "total number" are seeking numeric answers, questions beginning with "which year" are seeking date answers, and so on. A caveat here is that the network may have picked up correlations in the training dataset which would then get reflected in the attributions. This motivates having the human in the loop while synthesizing rules.
Axiomatic Attribution for Deep Networks
We apply our technique to word-level language modeling of the Penn Treebank dataset (Marcus et al., 1993), and apply an LSTM-based sequence model based on (Zaremba et al., 2014). For such a network, given a sequence of input words, and the softmax prediction for the next word, we want to identify the importance of the preceding words.
Axiomatic Attribution for Deep Networks
To describe the setup, we choose 20 randomly chosen sections of the test data, and for each of them inspect the pre- Figure 4: Attributions from question classification model. Term color indicates attribution strength-Red is positive, Blue is negative, and Gray is neutral (zero). The predicted class is specified in square brackets.
Axiomatic Attribution for Deep Networks
diction score of the next word using the first 10 words. The baseline here is obtained by zeroing out the embedding vectors, just as in the question classification model.
Axiomatic Attribution for Deep Networks
In Table 5 we show a comparisons of integrated gradients to gradients. Due to saturation, the magnitudes of gradients are so small compared to the prediction scores that it is difficult to make sense of them. In comparison, integrated gradients have a total amount close to the prediction, and seem to make sense. In the example, in predicting the second "ual", integrated gradients are clearly the highest for the first occurrence of "ual", which is the only word that is highly predictive of the second "ual".
Axiomatic Attribution for Deep Networks
We apply integrated gradients to a network performing Ligand-Based Virtual Screening which is the problem of predicting whether an input molecule is active against a certain target (e.g., protein or enzyme). In particular, we consider a network based on the molecular graph convolution architecture proposed by (Kearnes et al., 2016). Understanding feature importance in molecules predicted to be active against may help medicinal chemists in identifying the dominant molecular substructures-formally, pharmacophores-that are responsible for the activity. It may also help sanity check the behavior of the network-we discuss an anecdote about this later in this section.
Axiomatic Attribution for Deep Networks
The network requires an input molecule to be encoded by hand as a set of atom and atom-pair features describing the molecule as an undirected graph. Atoms are featurized using a one-hot encoding specifying the atom type (e.g., C, O, S, etc.), and atom-pairs are featurized by specifying either the type of bond (e.g., single, double, triple, etc.) between the atoms, or the graph distance between them. The baseline input is obtained zeroing out the feature vectors for atom and atom-pairs.
Axiomatic Attribution for Deep Networks
We visualize integrated gradients as heatmaps over the the atom and atom-pair features with the heatmap intensity depicting the strength of the contribution. Figure 6 shows the visualization for a specific molecule. Since integrated  Figure 5: Prediction for ual: 0.0062, total integrated gradient: 0.0063 gradients add up to the final prediction score (see Proposition 1), the magnitudes can be use for accounting the contributions of each feature. For instance, for the molecule in the figure, atom-pairs that have a bond between them cumulatively contribute to 46% of the prediction score, while all other pairs cumulatively contribute to only −3%.
Axiomatic Attribution for Deep Networks
We presented the attributions for 100 molecules active against a specific task to a few chemists. The chemists were able to immediately spot dominant functional groups (e.g., aromatic rings) being surfaced by the attributions. Figure 6: Attribution for a molecule under the W2N2 network (Kearnes et al., 2016). The molecules is active on task PCBA-58432.
Axiomatic Attribution for Deep Networks
Identifying Degenerate Features. One application of an attribution method is to debug and improve a deep network. We now discuss how attributions helped us spot an anomaly in the W1N2 architecture in (Kearnes et al., 2016). On applying the integrated gradients method to this network, we found that several atoms in the same molecule received identical attribution despite being bonded to different atoms. This is surprising as one would expect two atoms with different neighborhoods to be treated differently by the network.
Axiomatic Attribution for Deep Networks
On investigating the problem further we found that since the W1N2 network had only one convolution layer, the atoms and atom-pair features were not fully convolved. This caused all atoms that have the same atom type, and same number of bonds of each type to contribute identically to the network.
Axiomatic Attribution for Deep Networks
Despite the aforementioned problem, the W1N2 network had good predictive accuracy. One hypothesis for this is that the atom type and their neighborhoods are tightly correlated; for instance an outgoing double bond from a Carbon is always to another Carbon or Oxygen atom. As a result, given the atom type, an explicit encoding of the neighborhood is not needed by the network. This also suggests that equivalent predictive accuracy can be achieved using a simpler "bag of atoms" type model.
Axiomatic Attribution for Deep Networks
We already covered closely related work on attribution in Section 2. We briefly mention other related work.
Axiomatic Attribution for Deep Networks
Over the last few years, there has been a vast amount work on demystifying the inner workings of deep networks. Most of this work has been on networks trained on computer vision tasks, and deals with understanding what a specific neuron computes (Erhan et al., 2009;Le, 2013) and interpreting the representations captured by neurons during a prediction (Mahendran & Vedaldi, 2015;Dosovitskiy & Brox, 2015;Yosinski et al., 2015). In contrast, we focus on understanding the network's behavior on a specific input in terms of the base level input features. Our technique quantifies the importance of each feature in the prediction.
Axiomatic Attribution for Deep Networks
One approach to the attribution problem proposed first by (Ribeiro et al., 2016a;b), is to locally approximate the behavior of the network in the vicinity of the input being explained with a simpler, more interpretable model. An appealing aspect of this approach is that it is completely agnostic to the implementation of the network and satisfies implemenation invariance. However, this approach does not guarantee sensitivity. There is no guarantee that the local region explored escapes the "flat" section of the prediction function in the sense of 2. The other issue is that the method is expensive to implement for networks with "dense" input like image networks as one needs to explore a local region of size proportional to the number of pixels and train a model for this space. In contrast, our technique works with a few calls to the gradient operation.
Axiomatic Attribution for Deep Networks
The primary contribution of this paper is a method called integrated gradients that attributes the prediction of a deep network to its inputs. It can be implemented using a few calls to the gradients operator, can be applied to a variety of deep networks, and has a strong theoretical justification.
Axiomatic Attribution for Deep Networks
A secondary contribution of this paper is to clarify desirable features of an attribution method using an axiomatic framework inspired by cost-sharing literature from economics. Without the axiomatic approach it is hard to tell whether the attribution method is affected by data artifacts, network's artifacts or artifacts of the method. The axiomatic approach rules out artifacts of the last type.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Deep learning (DL) is having a major impact on the study of genetics owing to its impressive performance; however, because of its complex nature, it is often perceived as a 'black box' . With increasingly large amounts of data being generated at low cost and complementary hardware advances, the new forms of artificial intelligence (AI) are enabling diverse prediction tasks ranging from annotation of the regulatory genome to categorization of single-cell data [1][2][3][4] . By understanding the properties in large-scale data that are the basis for such successful predictions, it is anticipated that we can obtain deeper insights into the biological processes being studied. Over the past few years, new model interpretation techniques that allow such insights have been emerging rapidly in the field of explainable AI (xAI) 5 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
Deep learning models can learn complex patterns. In determining which DNA sequences in the human genome are directing gene transcription in a specific cell type, for instance, models may be learning sequence composition (for example, GC content), the presence or absence of specific motif patterns (transcription factor (TF) binding sites), the local accessibility of chromatin, biophysical properties of DNA (for example, bendability), positional differences between sequence properties (that is, features found in the edges or middle of the region), positive or negative interactions between features, or attributes that are beyond our current state of knowledge. To learn a large and complex feature set, such models learn millions of parameters that together determine the model predictions, but without providing an explanation about how a given prediction was made. Hence, the challenge of interpreting a complex dataset is essentially converted into a challenge of interpreting a complex model, with the benefit that probing the model is limited only by computation cost.
Obtaining genetics insights from deep learning via explainable artificial intelligence
In this Review, we present an organized overview of the key interpretation approaches with the intention of empowering researchers working across topics in genetics to incorporate xAI into their studies. We focus on the task of post-hoc interpretation, a popular class of xAI algorithms that are being increasingly used and improved for biological applications. Post-hoc interpretation operates on a trained model and aims to identify relevant combinations of input features and to quantify the importance of each feature -or combination thereof -to the model's performance. These algorithms thus provide a new and exciting approach for systematic hypothesis generation, aiding researchers to prioritize experimentation. Post-hoc interpretation methods can be divided into two types: local and global. Local interpretation aims to identify important features that affect a model's prediction on a single input example (for example, an individual DNA sequence). Global interpretation applies across all input examples to identify the combination of features that affects the model's overall performance.
Obtaining genetics insights from deep learning via explainable artificial intelligence
In the sections that follow, we first provide fundamental information on how deep learning methods are used in regulatory genomics, and then we categorize four approaches for interpretation: model-based Features Scalar inputs to a machine learning model.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The task of understanding a model's prediction for a single input.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The task of understanding how a model makes predictions across all inputs.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Obtaining genetics insights from deep learning via explainable artificial intelligence interpretation, mathematical propagation of influence, identification of interactions between features and use of prior knowledge for transparent models (FiG. 1). For each approach, we provide an intuitive understanding of how it works and explain its underlying assumptions and limitations in the context of typical high-throughput biological datasets. When applicable, we highlight how local interpretations can be aggregated to reveal a global understanding of the behaviour of a model, but we note that at the time of writing, there is no general framework for aggregating information from local interpretation techniques. Although we present methods with reference to successful cases, we recognize that the insights derived from the application of xAI methods are highly dependent on experimental design, dataset properties, and the accuracy of the trained models. Throughout the Review, we use examples from the field of regulatory genomics (see Box 1 for an introduction to gene regulation research), but the concepts presented can be generalized across a broad range of applications.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Neural networks and sequence-to-activity models. Deep neural network (DNN) models have emerged as the leading type of predictive model in regulatory genomics 1,4,6 . For this Review, we focus on sequence-to-activity models based on neural networks. These models take a putative regulatory DNA sequence (usually 100-10,000 bp) as input and aim to predict some dynamic property (that is, cell or context specificity) of the sequence's activity.
Obtaining genetics insights from deep learning via explainable artificial intelligence
For example, a model may predict whether a given TF binds to that sequence in a given cell type as measured by a chromatin immunoprecipitation followed by sequencing (ChIP-seq) experiment [7][8][9][10] . Other common prediction targets include chromatin accessibility 11,12 , RNA binding 13 , gene expression [14][15][16][17] , splicing 18,19 or aspects of chromatin 3D organization 20 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
The most widely used architectures for sequenceto-activity models are convolutional neural networks (CNNs) and recurrent neural networks (RNNs) 2 , both of which were initially developed for tasks in computer vision and natural language processing (see Box 2 for an overview of neural networks). CNNs include initial layers composed of convolutional nodes (also called filters), in addition to fully connected nodes in later layers.
Obtaining genetics insights from deep learning via explainable artificial intelligence
A convolutional node is a type of neural network node that serves as a pattern detector. It is scanned across the input, at each position evaluating whether the sequence matches a specific pattern, usually of length 3-20 bp. RNNs are composed of nodes arranged in a chain. In sequence-to-activity models, this chain is oriented along the input DNA sequence. Each node (or collection of nodes) in the chain takes a single DNA letter as input and outputs a value to the next node in the chain. Recurrent and convolutional architectures are often used together, such that sequence is passed through one or more convolutional layers before entering a recurrent layer 12,21 . No consensus currently exists regarding how to design the best neural network architecture for a given task, so researchers typically experiment with multiple architectures 21 . We point readers to previous review articles for a more complete discussion of various neural network architectures and their properties in the context of genomics datasets 1,2,4 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
To learn complex combinations of predictive features from data, state-of-the-art DNN models typically consist of tens of millions of free parameters that are learned during training. Although the high capacity of DNNs to encode latent feature representations results in state-of-the-art prediction accuracy, it comes with the challenge of identifying what features and feature combinations have been learned by the model. xAI algorithms can examine the inner workings of black box models such as DNNs to reveal the basis on which predictions are made. Intuitively, almost all post-hoc model interpretation methods can be understood through a feature removal recipe, whereby trained models are probed to assess the importance of a given feature or feature combination to the model's performance 22 . From a mathematical perspective, the challenge of deep learning model interpretation stems from the need to somehow navigate a vast combinatorial search space. Many frameworks have been proposed, including gradient-based 23 , perturbation-based 24 and game theory-based analysis 25,26 , to intelligently evaluate the combinatorial search space to produce local interpretations. However, each of these strategies comes with assumptions and limitations, and hence there is no globally optimal strategy for model interpretation. The second challenge is in generalizing from local interpretations, which estimate feature importance on a given input example at a time, to a global understanding of important feature combinations on an entire data corpus. Another key difficulty is the inability to systematically evaluate interpretation strategies, first because there is a lack of benchmark datasets in which the true set of important features is known ahead of time, and second because the validity of the assumptions made by the various algorithms depends on both the properties of the input dataset and the biological processes involved.
Obtaining genetics insights from deep learning via explainable artificial intelligence
An intuitive approach to model interpretation is to examine individual components of a network to understand what (hidden) patterns they represent and their These authors jointly supervised this work: maxwell W. libbrecht, Wyeth W. Wasserman, Sara mostafavi.
Obtaining genetics insights from deep learning via explainable artificial intelligence
A class of learning tasks that takes a DNA sequence as input and predicts a property of the activity of that sequence, such as transcription factor binding or chromatin accessibility in a cell type of interest.
Obtaining genetics insights from deep learning via explainable artificial intelligence
(CNNs). A neural network architecture that includes convolutional nodes.
Obtaining genetics insights from deep learning via explainable artificial intelligence
(RNNs). A type of neural network architecture in which nodes are arranged in a chain along a sequential input such as a DNA sequence.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Sets of neural network nodes that take input from nodes of the previous layer and output to nodes of the subsequent layer.
Obtaining genetics insights from deep learning via explainable artificial intelligence
(Also known as filters). A type of neural network node that takes input from a short contiguous sequence of nodes, usually 3-20 bp in sequence-to-activity models.
Obtaining genetics insights from deep learning via explainable artificial intelligence
(Also known as units and artificial neurons). The basic units of a neural network. They take input from other nodes and output scalar values to other nodes.
Obtaining genetics insights from deep learning via explainable artificial intelligence
www.nature.com/nrg contributions to the prediction performance. Although this can be effective for small networks, modern DNNs are prohibitively large and complex. However, even for deep networks, often a partial model-based interpretation can be helpful in extracting important features learned by the model. For instance, in computer vision, artificial neurons in hidden layers have the potential to learn semantically interpretable features with increasing levels of abstraction 27 Fig. 1 | Conceptual approaches to explainable artificial intelligence. a | A schematic representation of a sequenceto-activity convolutional neural network. The input DNA sequence is one-hot encoded and scanned by convolutional nodes in the first layer (cyan). The activation values of convolutional nodes get passed through non-linearities introduced in subsequent layers (grey). The output layer (red) predicts a property of interest. b | Model-based approaches focus on annota ting individual parts of the model. For example, a convolutional node (cyan) represents a position-weight matrix (red heat-map) and can be linked to a transcription factor (TF) binding site or other sequence pattern. c | Propagation of influence predicts the importance of each input feature (for example, base pair (bp)) according to the gradient of a neural network model (∇f). Subsequences of important base pairs correspond to features that are informative to the model's prediction. d | As these networks learn a nonlinear function, they can be studied for the presence of interactions between input features. Interaction identification methods can output the interaction strength of each pair of input features. In the example shown, the pairwise interactions between nucleotides in the input DNA sequence are indicated by the intensities in the heat-map plot. e | The original network can be designed to reflect biological concepts, which also makes it transparent. Researchers use prior knowledge from databases including those of motifs or Gene Ontology subsystems.
Obtaining genetics insights from deep learning via explainable artificial intelligence
presence of edges in input images at various orientations on a fine scale, and then the deeper layers use combinations of edge detectors to identify higher-level image parts such as ears, nose and mouth 28 . Analogously, in sequence-to-activity models, first-layer neurons may learn short subsequences that represent TF binding motifs, and the deeper layers construct various combinations of these motifs as composite features that are useful for prediction of cell-type-specific chromatin states 11 . Two main method categories for model-based interpretation can be considered. The simplest approach directly examines the activity of hidden neurons to extract a set of relevant features. The second approach initially trains models with an attention mechanism, which directly yields a measure of relevance per input feature through a set of learned attention weights. In this section, we describe how these approaches can be applied to sequence-to-activity models to understand individual features that are learned by the first layer of the network, and in the 'Identifying interactions between features' section (below) we describe approaches to identify combinations of features learned in deeper network layers.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Interpreting first-layer convolutional nodes. In a convolutional sequence-to-activity model, first-layer neurons (filters) capture short sequence motifs, encoded in convolutional weight matrices. Mathematically, the operations performed by application of convolutional weight matrices to sequence are equivalent to scanning the sequence with a position-weight matrix (PWM) (FiG. 2a). Thus, one can apply a simple transformation (called softmax) to a convolutional weight matrix to derive a position frequency matrix (PFM) 13 that quantifies position-dependent nucleotide frequencies, and further scaling and log-transformation enables visualization of the matrix associated with a filter as a standard PWM 29 (which quantifies the log likelihood of each nucleotide in the motif represented by the corresponding convolutional matrix). However, unconstrained learning of weight values may create scaling issues that render this approach less effective. In practice, a more common strategy is to search for subsequences that activate a given filter above a chosen threshold and directly construct a PWM based on the alignment of the set of activating subsequences (FiG. 2b). One can use the entire input dataset to search for maximally activating subsequences, or solve a more general optimization problem that searches among all possible subsequences of length m for those that maximally activate a given filter 30,31 . These PWMs can then be annotated by comparing them with known TF binding profiles in databases such as JASPAR and Cis-BP 32,33 . Although there is no guarantee that CNN filters resemble known TF binding motifs, previous work has shown that the learned PWMs typically do so. For example, when CNNs were applied to assay for transposase-accessible chromatin sequencing (ATAC-seq) data across various immune cells, most learned PWMs mapped to known TF binding motifs that have prominent roles in immune cell differentiation, including motifs for PAX5, EBF1 and LEF1 (ReF. 11 ). Other studies have shown similar results 8,12 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
Because neural networks are over-parameterized by design, the mere presence of a PWM does not imply that it is a predictive, interesting or useful feature. Therefore, we need to measure the contribution of the PWM to the model's predictions 8,11,18 . In the node-based strategy, this is achieved by nullifying (or ablating) each filter in turn and measuring the impact of such nullification on the model's predictions (FiG. 2c). Intuitively, if an influential filter is nullified, then the network's prediction(s) should be significantly altered. The impact of filter nullification can be measured for each input example, giving a local interpretation. To form a global interpretation, the simplest approach is to average the local interpretations.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Node-based strategies are reductionist approaches to understanding a complex system, with the core assumption that individual units are independently interpretable on their own. However, because DNNs are usually trained to be robust to drop-out 34,35 of individual neurons, in practice, an important pattern may be captured
Obtaining genetics insights from deep learning via explainable artificial intelligence
Region in genomic DNA that can contribute to gene regulation.
Obtaining genetics insights from deep learning via explainable artificial intelligence
A component of a neural network that can learn to adaptively prioritize (that is, pay attention to) certain parts of an input by weighting.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Weights learned by the attention mechanism.
Obtaining genetics insights from deep learning via explainable artificial intelligence
A form of regularization typically used during training of neural networks in which activations from subsets of hidden units are zeroed out.
Obtaining genetics insights from deep learning via explainable artificial intelligence
understanding the control of gene transcription is crucial to understanding many aspects of cellular biology. multiple biochemical mechanisms influence gene transcription, including the structure of chromatin, the binding of transcription factors (TFs) to DNa, the components of the rNa polymerase (rNa Pol) complex and the actions of proteins that interact with the above 87 (see the figure). at a higher level, studies explore the different types of regulatory element within the genome such as promoters, from which rNa transcription is initiated, and distal regulatory elements such as enhancers, which confer patterns of specificity upon promoters 88,89 . owing to the crucial role of gene regulation in controlling the phenotype of cells 90 , new technologies for measuring cellular properties related to gene regulation are constantly emerging, providing higher resolution and broader coverage, which have transformed the data landscape of the field. at the core of experimental studies are assays that quantify the structure of chromatin (for example, Hi-C) 91 , the accessibility of chromatin (for example, DNase hypersensitivity, and assay for transposase-accessible chromatin sequencing (aTaC-seq)) 89 , and the presence of proteins or modified forms of proteins at specific locations along the DNa 92 (for example, chromatin immunoprecipitation followed by sequencing (ChIP-seq) for TFs or histone modifications, or many specialized forms of ChIP-seq, such as ChIP-exo). as measurements across an entire tissue capture heterogeneous mixtures of cell types, these technologies are increasingly applied to isolated cell types and more recently at single-cell level. Deep learning methods have been propelled to the forefront of the analysis of such data as their size and complexity continue to expand 1,3,4,93,94 . Common questions pursued by researchers include the identification of and interplay between regulatory sequences within the genome that are active in a particular cell or cell context 7,8,11 , the locations of TF binding sites within such regions, the identity of and cooperative interplay between TFs to derive a given cell state 16 , the 3D structure of chromatin in the nucleus and much more 20  multiple times by different neurons: that is, nodes may be redundant. In this situation, nullifying a single neuron will not provide the true importance of a pattern to the model's predictions. Additionally, depending on the architecture of the model, a biologically interpretable pattern may be learned as a combination of subcomponent nodes. For example, a long TF motif may be learned by two filters, each learning some part of the motif 36 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
Attention mechanism weights for visualizing feature importance. Weight regularization is a general technique to help mitigate the issue of overfitting in training neural networks and often improves both performance and ability to isolate important features 35 . The simplest strategy for weight regularization is to add a term to the objective function during model training to encourage the learned weights to have certain properties, for example, to be small in magnitude or to be sparse with few non-zero elements 37,38 . However, one can also consider different types of weighting in neural network models. Attention (and its many variants) can be considered a form of weight regularization for which weights are introduced for input sequences to prioritize positions in the input as likely to hold relevant information for processing 39 . The computation of these weights is generally handled by an additional module in the network that automatically learns the attention weights during training to capture relevance between hidden states in machine learning tasks 40 . Attention mechanisms can improve both performance and interpretability of neural network models 41 , and in particular address the challenge of performance degradation with respect to increasing input sequence length for RNNs 39 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
As the name implies, when combined with model training, attention weights force the model to focus on a limited portion of the input while learning hidden features. In the case of sequence-to-activity models, the attention vector can be examined directly to aid in Overfitting The case when a machine learning model is specific to its training set and does not generalize to other inputs.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The target outputs of a classification model.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The process of converting a DNA letter into a length-4 vector such that one position is set to 1 and the others are set to 0, for use as input to a neural network.
Obtaining genetics insights from deep learning via explainable artificial intelligence
a neural network is a model that implements a mathematical function taking input features (such as a DNa sequence) and producing one or more output labels (such as predicted transcription factor (TF) binding or gene expression) 35 . Neural network models are constructed from many nodes, also known as units or artificial neurons, organized in layers (see the figure). For sequence-to-activity models, input DNa sequences of length L are converted using one-hot encoding: a binary matrix of size 4 × L, where each column has a single entry set to 1 indicating the DNa base (a, C, T or G) and all other entries set to 0. Neural network models taking length L one-hot encoded sequences have 4L input nodes. Internal nodes compute their values as σ(w 1 x 1 + ⋯ + w n x n + b), where (x 1 ,…,x n ) are node inputs, (w 1 ,…,w n ) are the weights, b is the bias and σ is the activation function. The output node of a neural network represents its prediction. Some neural networks have multiple output nodes, and thus solve multiple tasks at once; for example, sequence-to-activity models may simultaneously predict the binding of multiple TFs, or the same TF across multiple cell types. The training process adjusts the weights (w 1 ,…,w n ) and bias (b) of each node to maximize agreement between the output nodes and labels of a training set.
Obtaining genetics insights from deep learning via explainable artificial intelligence
one can construct single-layer neural networks that are mathematically equivalent to linear or logistic regression, depending on the activation function. Deep neural networks (DNNs) contain additional layers of nodes other than the input and output; these nodes are hidden nodes because their values are never observed directly but instead are learned from data during training. The non-linearity of DNNs stems from the fact that they use nonlinear activation functions in at least some of the layers. a popular activation function is the rectified linear unit (relu). a pooling operation is typically applied to the output of a convolutional layer (that is, activation map), whereby the maximum or average of nearby elements is used to reduce the number of learned parameters in the later layers. Finally, a flattening operation combines multiple activation maps into a single vector to use as input to the fully connected layer. a DNN model with only linear activation functions would still be equivalent to linear regression. The neural network architecture refers to the number of hidden nodes and how they are connected to each other. DNNs are typically organized into layers, where the nodes of each layer take the previous layer nodes as input. The objective function is the mathematical function that the training process seeks to optimize; for example, mean squared error (m.s.e.) is the objective function typically used to train a model on continuous output. recent reviews provide a deeper overview of machine learning methods, including DNNs, and their application to high-throughput biological datasets 1,3,4,95 . Finding subsequences that activate the given nodes above a threshold enables researchers to annotate them through PWM construction. c | Ablating a filter or node shows how important that node is for the final prediction. d | Attention is an extra layer in the network that helps the network to focus only on the most relevant input features. Visualizing attention weights gives an impression of important features that are used to make predictions.
Obtaining genetics insights from deep learning via explainable artificial intelligence
www.nature.com/nrg identifying parts of the input that played a key part in the model's internal representation (FiG. 2d).
Obtaining genetics insights from deep learning via explainable artificial intelligence
Previous empirical work has shown some promise in directly interpreting attention weights in the sequence-to-activity setting. For example, attention weights were shown to correlate with the expected location of the TF binding site in the sequence 42 , and coincide with DNase footprint locations 43 . However, other studies caution against direct interpretation of attention weights 44 . For example, redundancy and multicollinearity in input features, which are typically abundant in high-throughput data, can result in instability in the estimation of attention weights.
Obtaining genetics insights from deep learning via explainable artificial intelligence
In contrast to probing components of a trained neural network, another class of algorithms operates directly on input examples by propagating perturbed data through the model and observing the effect on predictions. These propagation-based feature attribution methods are thought to be more model-agnostic, as to some extent they can bypass the side-effects of specific model architectures (for example, redundancies in learned filters) when determining feature importance. They also solve the problem of feature identification and quantification of the importance of each feature simultaneously. Propagation-based attribution methods can be divided into two major groups: forward and backward. In this section we discuss the conceptual foundations behind the most popular attribution methods, their shortcomings and mitigation strategies in the context of genetic data.
Obtaining genetics insights from deep learning via explainable artificial intelligence
In computer vision, perturbation strategies have been used for decades. The simplest form is pixel flipping, whereby one or multiple pixels of an input image are modified to identify features in the image that are relevant for the prediction results 45 . If changing the value of a pixel has a large effect on the classification result, then it may correspond to a feature (or part of a feature) that the model has identified as relevant for making predictions. Analogous to image pixel flipping one can consider flipping elements corresponding to nucleotides of biological sequences to determine feature importance for trained models (FiG. 3a). This strategy is termed in silico mutagenesis (ISM) 7 , owing to the parallels between this approach and in vitro mutagenesis of DNA.
Obtaining genetics insights from deep learning via explainable artificial intelligence
To perform ISM in practice, given an input DNA sequence X of length L, each index i (nucleotide) in the input sequence is selected, and for each of the three alternative nucleotides a new sequence is generated by changing only the entry in the ith position. The difference between the model's prediction on the alternative sequence and the original sequence is often referred to as the attribution score. Repeating for all nucleotides results in a 4 × L matrix called an attribution map 46 , which can be visualized as a sequence logo (FiG. 3a). The approach has been shown to deliver superior results to other attribution-based approaches 47,48 . Although ISM is typically used to record the perturbation effect on the final layer output, it can also be applied to discover attributions for hidden neurons.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Applying ISM at a single-nucleotide level across many input examples results in a large computational overhead (order of 3L per input sequence). To save computation, one can limit the analysis to a subset of input sequences of greatest expected insight value (such as sequences with strong predictions). Other strategies take advantage of specific architectures of neural networks to bypass some redundant computations, for example, only computing values in the receptive field of a given CNN convolution filter. These include fastISM 49 and accelerated ISM 50 . Applying these strategies enables efficient application of single-nucleotide-level ISM, but it is not computationally practical to exhaustively examine all possible combinations of nucleotides.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Instead of per-nucleotide ISM, larger stretches of the input sequence can be altered to identify important motifs that depend on a combination of important base pairs (akin to laboratory-based scanning mutagenesis) (FiG. 3b). Such forms of occlusion were used to find enhancer-gene pairs 16 , CTCF (CCCTC-binding factor) sites with certain orientations 20 and boundaries of cis-regulatory elements 51 . Instead of random occlusion, greater interpretability may be gained by mutating known TF motifs (also called motif mutagenesis), but this approach requires a priori knowledge of the motif sites. For instance, one can scan existing TF PWMs on a given sequence to get a per-nucleotide scanning score, and then use ISM to alter the sequence of high-scoring segments.
Obtaining genetics insights from deep learning via explainable artificial intelligence
This idea of partial occlusion can further be generalized to scanning models on synthetic sequences that contain specific subsequences (k-mers) at specific sequence positions 52,53 , to gain higher-resolution insight into sequence properties, including the importance of motif position, spacing or flanking sequences 54 , without having to rely on our incomplete knowledge about TF binding patterns. As one may expect on the basis of the underlying biology, the sequence context onto which such subsequences are positioned tends to have an impact on the results. Computationally, less-biased results may be obtained by preserving dinucleotide composition and motif positional information in the synthetic sequences 55,56 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
Forward propagation methods are computationally expensive owing to the large number of forward passes required to generate accurate statistics. Back-propagation methods were developed to address this issue. These methods approximate ISM by evaluating the derivative of the model F at a given input sequence to compute the impact of infinitesimally small changes to the sequence on the model's prediction 46 (FiG. 3c).
Obtaining genetics insights from deep learning via explainable artificial intelligence
As neural networks are constructed from the composition of multiple nonlinear functions, the gradient of the model F needs to be computed with the chain rule using the back-propagation procedure (backward propagation of partial derivatives through the network). This yields a function that then needs to be evaluated at a specific input to produce a gradient vector. The resulting vector has size equal to the number of input features. An element-wise product between the gradient Attribution score An importance score assigned to a given input feature by a post-hoc local interpretation method.
Obtaining genetics insights from deep learning via explainable artificial intelligence
(Also known as saliency map or relevance map). An estimate of how much each input feature contributes to the output, produced by certain local interpretation methods.
Obtaining genetics insights from deep learning via explainable artificial intelligence
vector for a given input and the input values produces a gradient-based attribution map (FiG. 3c). More recent approaches, such as GradCAM 57 and its variants 47 , attempt to improve the quality of gradient approximation through additional local weighting of feature importance.
Obtaining genetics insights from deep learning via explainable artificial intelligence
A major shortcoming of gradient-based methods is the so-called saturation problem. Redundancy in input features can result in underestimation of importance values. For instance, a regulatory sequence may contain multiple copies of the same binding motif. The overall sensitivity of the model is distributed across the multiple copies. Reference-based gradient methods, which include the widely used integrated gradients 23,58 , are designed to address this problem. Instead of evaluating the gradient at a single point, the integrated gradients method considers integrating the gradient of the model F along the line l α αX α X ( )= +(1 − ) where X represents a baseline (also called the reference) version of X and α ∈ [0, 1]. This procedure allows for a more sensitive assessment of input perturbation on model predictions. For genomic sequences, a mononucleotide or dinucleotide shuffled input sequence provides a reasonable baseline 59,60 . The integrated gradients method does not have a saturation problem because gradients are computed with respect to the baseline X, which, in the TF binding example, will lack both binding sites. An improved variant of integrated gradients, termed enhanced integrated gradients (EIG), was proposed to additionally consider all nonlinear paths between the baseline and the sample. EIG was benchmarked against various types of baseline, and used to identify A1CF as a novel regulator of the liver splicing programme 61 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
A general issue with gradient-based methods is numerical instability in computing gradients of DNN models via back-propagation. Also, even for a continuous neural network model F, the gradient of F may possess discontinuities, for example, if it is composed of rectified linear unit (ReLU) activations. This can result in large jumps in the gradient of F for small changes in the input, leading to unreliable gradient estimation. DeepLIFT is a back-propagation method that directly addresses this shortcoming 60 . DeepLIFT was previously used to reveal key markers of pluripotency 10 , and to   www.nature.com/nrg identify novel TF binding motifs that contribute to epidermis differentiation 9 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
Unlike integrated gradients, DeepLIFT operates on the differences between the input and the reference at every node in the network. Attribution scores approximating the gradient of F are obtained by back-propagating these differences in one pass from the output to input nodes. DeepLIFT has multiple rules for assigning attribution scores, one of which can be shown to be an approximation of the Shapley values 26 . Shapley values provide a sound mathematical framework for assigning feature importance proportional to their marginal contribution to a prediction result in cooperative game theory. They were adapted for deriving feature attributions and were shown to provide a theoretical basis for unifying several feature attribution algorithms including DeepLIFT and DeepSHAP 22,26,62 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
To generalize from per-sequence attribution maps that are produced by propagation-based methods to reveal a global understanding of important motifs, the results across many input examples need to be aggregated. TFMoDisco is one such approach that is specifically designed for DNA input sequences and relies on clustering across attribution maps to identify globally important sequence motifs 63 . As one example, TFMoDisco was used to identify sequence features that are important for telomerase reverse transcriptase (TERT) promoter function 64 . However, the typical challenges of clustering remain in this aggregation problem as well, including uncertainty in setting the number of clusters, defining similarity metrics and adjusting the resolution of the clustering. A few recent approaches have been specifically designed for aggregating the results of partial occlusion-based methods that provide additional statistical guarantees 65,66 . Koo and colleagues 65 showed that such methods can reveal sequence features that are important for predicting RNA-protein interactions.
Obtaining genetics insights from deep learning via explainable artificial intelligence
So far, we have highlighted approaches for identifying individually important features, such as TF motifs, that contribute to the model's prediction(s). However, the power of neural networks stems from the ability to model nonlinear interactions between features. In the context of gene regulation, it is widely recognized that interactions such as cooperativity between TFs can account for activity beyond that attached to each TF in isolation. Researchers are therefore motivated to detect interactions between features identified by the neural network that are likely to represent such cooperative TF behaviour. In this section, we describe how the methods for producing local interpretation described previously can explain interactions between features.
Obtaining genetics insights from deep learning via explainable artificial intelligence
As deeper layers of neural networks hierarchically assemble features learned in lower layers, one obvious strategy for identifying interactions is to examine deeper layer neurons. For instance, by examining sequences that activate second-layer filters, Bogard and colleagues 67 identified and validated interaction effects between RNA binding proteins involved in alternative polyadenylation. For this type of task in computer vision applications, optimization-based methods that search among a large set of random inputs (such as images) for those that maximally activate a given hidden neuron tend to work best 68 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
A model that uses self-attention, a popular variant of the general attention mechanism 40,69 , directly represents interactions between base pairs in a sequence-to-activity model 43,70 . Self-attention can be used to replace the pooling layer in a CNN model. In this layer, the model produces an L × L attention matrix that represents the attention paid between each pair of the L base pairs of the input sequence. This attention matrix can be used as a representation of cooperativity between base pairs. For example, in a model that aims to predict the expression of a gene from a wide (for example, 200 kb) region around a transcription start site, self-attention may represent enhancer-promoter interactions responsible for regulating that gene 16,71 . In another example, in a model that aims to predict TF binding, self-attention between pairs of positions in the input sequence may represent cooperative TF binding 16,72 or more generally highlight regions in input sequences that are important for TF binding 73 .
Obtaining genetics insights from deep learning via explainable artificial intelligence
ISM becomes prohibitively computationally expensive when used to estimate potential interactions. This is because a separate input must be tested for each pair of features, and thus the number of tests grows quadratically as the number of features increases linearly. However, ISM can be feasibly applied in a restricted fashion, for instance, for a targeted analysis of sequences that contain a specific motif pair 9 or more generally through insertion of two motifs into random sequence. In this setting, ISM mutates one motif -either a single position or the whole motif -and compares the resulting prediction with that from the intact sequence (FiG. 4a). Applying this approach allows for discovery of additive and non-additive effects between motifs, and also for assessment of the influence of motif spacing 9,54 . For example, to assess motif spacing dependencies, two features (TF motifs or k-mers) can be inserted into random genomic sequence, then one feature can be held in a fixed position (for example, at the centre of the sequence) while the second feature is slid iteratively across the sequence. This approach was effective in identifying position-dependent interactions between NANONG and SOX2 during stem cell differentiation 10 . This procedure can also be applied to two identical motifs to test multimerization. To overcome noise or statistical instability, the process can be applied to multiple sequences. Another variant of this idea involves combining forward and backward propagation methods, in an approach termed deep feature interaction maps (DFIM) 53 (FiG. 4b). DFIM represents a trade-off between forward and backward propagation algorithms,
Obtaining genetics insights from deep learning via explainable artificial intelligence
A type of attention mechanism in which every part of the input is compared with every other part, including itself.
Obtaining genetics insights from deep learning via explainable artificial intelligence
as it requires several passes through the network, which grows linearly in the number of features, in contrast to the quadratic growth of naive second-order ISM. In the DFIM approach, a feature interaction score (FIS) between any pair of features (source feature and target feature) in an input DNA sequence is computed as the change in the importance score of the target feature when the source feature is perturbed, while keeping all the other features in the sequence intact.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Back-propagation-based methods such as integrated gradients can also be extended to exhaustively assess pairwise feature interactions. Instead of first-order derivatives, one can assess the second-order derivative of the model's prediction, termed integrated Hessians 74 . This approach yields a matrix of second derivatives H, in which H(i, j) corresponds to the degree to which feature i modulates the influence of feature j or vice versa. A high value of H(i, j) indicates an interaction between these two features. Notably, integrated Hessians cannot be applied to neural networks that use an activation function without an informative second derivative, such as the widely used ReLU activation function 75 . Using integrated Hessians could plausibly enable efficient evaluation of interaction effects -for example, revealing the interactions between every pair of base pairs in a DNA sequence -but they have yet to be systematically tested for genomics applications.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Using prior knowledge for transparent models In comparison with the input nodes, interpreting hidden nodes in deeper layers is more challenging because each corresponds to a complex nonlinear function of the inputs and may not correspond to any observable quantity. A transparent neural network model is one in which the hidden nodes are constructed to physically correspond to biological units at a level of granularity that is helpful for human understanding 76 (FiG. 5).
Obtaining genetics insights from deep learning via explainable artificial intelligence
To build models that have inherently interpretable units, one needs to use prior knowledge to design the network architecture. For example, one can initialize filters according to known TF binding motifs 12 (FiG. 5a). The deeper layers construct combinations of these input features to model parts of biological systems at higher levels of abstraction. For example, the second layer may represent co-binding relationships between motifs, a higher layer may correspond to biological pathways and so on. Although prior knowledge about interactions between TFs and pathway memberships are sparse in general, previous work has used partial knowledge from sources such as Gene Ontology 77 to either hard-code these as binary edges between neurons or more softly encourage edges between units across different layers through regularization 78 . Inspecting a model trained in this way can grant insight into the presence of such prior interactions in a given context (FiG. 5b). Although building 'transparent' models, such as probabilistic graphical models, was a focus before the recent popularity of neural networks, there are two reasons that can motivate the creation of such models with neural networks. First, accompanying optimization algorithms can be used to learn nonlinear interactions between interpretable units, and second, computational infrastructure can be re-used to optimize and apply models on large datasets that would not be feasible otherwise.
Obtaining genetics insights from deep learning via explainable artificial intelligence
One of the first transparent neural network models in biology to use neural network computational 
Obtaining genetics insights from deep learning via explainable artificial intelligence
Observed effect ∇f Fig. 4 | Approaches to reveal interactions between features in model performance. a | A scheme of the combinatorial motif in silico perturbation analysis. A given sequence with two motif instances is changed so that either one or both motifs are mutated. The network's output is recorded for each alternative and compared with the reference. The prediction change for the joint mutation is compared with the sum of changes for the marginal mutations to infer the potential interaction effect. b | Deep feature interaction maps (DFIMs) compare the attribution scores computed for the reference sequence with both motif instances with the scores of an altered sequence, in which one of the motifs is scrambled. If the scores differ significantly, it might imply dependency of the motifs on each other.
Obtaining genetics insights from deep learning via explainable artificial intelligence
A function applied to the output of neurons, typically to model non-linearity.
Obtaining genetics insights from deep learning via explainable artificial intelligence
A common machine learning scheme that controls model expressivity by including a term in the objective function that penalizes model complexity.
Obtaining genetics insights from deep learning via explainable artificial intelligence
www.nature.com/nrg infrastructure is called DCell 76 . DCell models the relationship between genotype and growth rate in yeast.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The structure of this model consists of first-layer nodes that represent genes and second-layer nodes that are constructed as functional groups defined hierarchically according to the Gene Ontology database. The model's transparency assisted in explaining the link between mutations that disrupt the genes PMT1 and IRE1 that impact growth rate. Both genes were linked to more than 200 Gene Ontology subsystems, but only the endoplasmic reticulum unfolded protein response node was substantially affected compared with the wild type. Experimental validation confirmed the correlation between this subsystem and the double mutants. More recently, the P-NET 79 model was developed to create transparent representations of molecular variables from multi-omic datasets. The model's transparency helped the authors to discover novel biomarkers (such as MDM4) of metastatic prostate cancer. In a similar spirit to DCell and P-NET, others have created neural networks in which units and parameters have a biophysical interpretation 80,81 . Recently Agarwal and colleagues 82 introduced a simplified neural network architecture called neural additive models (NAMs), which sacrifice some of the capacity of traditional neural network models to gain direct parameter interpretation. Although they are only able to detect linear relationships between features (which can still be learned de novo), their early application to genomics shows competitive performance relative to more complex models in some settings 83 . However, the main drawback is the inability of such models to learn potential interactions between features (such as nonlinear relationships between pairs of motifs, or dependence of motif effect on spacing and flanking sequence).
Obtaining genetics insights from deep learning via explainable artificial intelligence
The primary limitation of this transparent modelling approach is that it requires the user to have systematic prior information. It may not be directly applicable to tasks for which the entities or their hierarchical structure are poorly characterized, or to tasks for which such entities are not assayable. The user must also take caution in interpreting the nodes of transparent models, because the approaches described here may fail to enforce the equivalence between nodes and biological entities. For example, a CNN initialized with TF motifs may see its filters drift away from those motifs during training. Also, techniques for enforcing transparency can degrade performance. For example, hard-coding of pairwise edges to encode TF-TF interactions or genepathway memberships are likely to reduce accuracy relative to an unconstrained model (although this reduction may be less severe when using softer regularization techniques).
Obtaining genetics insights from deep learning via explainable artificial intelligence
In this Review we focus on sequence-to-activity models, yet the described xAI methods apply broadly across applications of deep learning in genomics, including to models of phenotype, gene expression and other multi-omics measurements, and single-cell measurements. However, interpretation may be more challenging in cases in which, unlike sequence models, the direction of causality is unclear.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The goal and utility of model interpretation depend strongly on the target application, which should therefore guide the choice of interpretation approach. When a user needs an overall understanding of a whole biological process, such as when aiming to make experimentally testable mechanistic hypotheses, a global interpretation a | Initializing first-layer convolutional filters (cyan circles) with known motifs from existing databases helps to make convolutional neural networks (CNNs) internally more interpretable. b | Transparent models are built using biological prior knowledge about gene regulatory hierarchies and functional annotations. Models are typically initialized with sparse edges (solid lines) representing biologically established connections (for example, a pathway and its gene member). The model may learn additional connections (dashed lines) representing novel associations. TF, transcription factor. approach is appropriate. When a user needs to understand an individual prediction made by a model, such as when identifying genetic risk of disease, a local interpretation approach is appropriate.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Interpretation methods ultimately amount to identifying and extrapolating from predictive patterns discovered within the training data. Spurious correlations present in the training data can present themselves in interpretation of the downstream model. In particular, when the model is trained on a dataset that is small or contains artefacts, a predictive model can achieve high accuracy by learning non-biological 'shortcuts' , and thus interpretation will not yield meaningful biology. Interpretations can provide a way to identify such spurious correlations; for example, in a medical imaging study, model interpretation enabled the researchers to identify hidden batch effects that were being used by the model to make predictions 84 . Thus, interpretation can also be a tool for debugging predictive models.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The quality of biological insight granted by model interpretation depends crucially on the prediction accuracy of the interpreted model. For example, when performing ISM, a perfect model would by definition yield the correct consequence of a given mutation, whereas a low-accuracy model may yield spurious inferences. Interpretations should be used with care when approaching tasks in which state-of-the-art prediction accuracy is far from perfect, such as when predicting gene expression or phenotype. This problem is particularly acute when interpreting higher-order effects of a model such as feature interactions, because any combination of feature values occurs in just a small number of training examples.
Obtaining genetics insights from deep learning via explainable artificial intelligence
The reliability of interpretation is hampered by the 'un-identifiability' of the interpreted model 85,86 . That is, because DNN models typically contain far more parameters than training examples, the training procedure is not guaranteed to find the best possible model. This means that the model parameters are sensitive to random selection of training examples and the initialization parameters. Model-based interpretations are most sensitive to this un-identifiability issue; however, we believe this phenomenon affects all interpretation techniques to varying degrees. Thus, interpretation must be used with care and with the understanding that some properties of a model, and the resulting features, may arise owing to chance. This issue can be alleviated by comparing multiple datasets and multiple training initializations, but this is sometimes too costly to perform.
Obtaining genetics insights from deep learning via explainable artificial intelligence
At the time of writing, there is no consensus regarding which xAI approaches are most effective. Part of this variability stems from the variety of goals held by users of these approaches. However, even when the goal is fixed, different approaches can each yield different insights. Interpretation methods sometimes yield results that are biologically nonsensical, perhaps for the reasons above or others that are not yet understood. We expect that as this field matures, best practices will become established and integrated into accessible analysis tools.
Obtaining genetics insights from deep learning via explainable artificial intelligence
As the size and availability of biological datasets grows, it becomes more important to investigate complex relationships between features using models. Distilling insight from these models requires effective xAI methods. Thus, xAI will have an ever-more central role in genomics.
Obtaining genetics insights from deep learning via explainable artificial intelligence
Published online 3 October 2022
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
With the deep Convolutional Networks (ConvNets) [10] now being the architecture of choice for large-scale image recognition [4,8], the problem of understanding the aspects of visual appearance, captured inside a deep model, has become particularly relevant and is the subject of this paper.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
In previous work, Erhan et al. [5] visualised deep models by finding an input image which maximises the neuron activity of interest by carrying out an optimisation using gradient ascent in the image space. The method was used to visualise the hidden feature layers of unsupervised deep architectures, such as the Deep Belief Network (DBN) [7], and it was later employed by Le et al. [9] to visualise the class models, captured by a deep unsupervised auto-encoder. Recently, the problem of ConvNet visualisation was addressed by Zeiler et al. [13]. For convolutional layer visualisation, they proposed the Deconvolutional Network (DeconvNet) architecture, which aims to approximately reconstruct the input of each layer from its output.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
In this paper, we address the visualisation of deep image classification ConvNets, trained on the large-scale ImageNet challenge dataset [2]. To this end, we make the following three contributions. First, we demonstrate that understandable visualisations of ConvNet classification models can be obtained using the numerical optimisation of the input image [5] (Sect. 2). Note, in our case, unlike [5], the net is trained in a supervised manner, so we know which neuron in the final fully-connected classification layer should be maximised to visualise the class of interest (in the unsupervised case, [9] had to use a separate annotated image set to find out the neuron responsible for a particular class). To the best of our knowledge, we are the first to apply the method of [5] to the visualisation of ImageNet classification ConvNets [8]. Second, we propose a method for computing the spatial support of a given class in a given image (image-specific class saliency map) using a single back-propagation pass through a classification ConvNet (Sect. 3). As discussed in Sect. 3.2, such saliency maps can be used for weakly supervised object localisation. Finally, we show in Sect. 4 that the gradient-based visualisation methods generalise the deconvolutional network reconstruction procedure [13].
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
ConvNet implementation details. Our visualisation experiments were carried out using a single deep ConvNet, trained on the ILSVRC-2013 dataset [2], which includes 1.2M training images, labelled into 1000 classes. Our ConvNet is similar to that of [8] and is implemented using their cuda-convnet toolbox 1 , although our net is less wide, and we used additional image jittering, based on zeroing-out random parts of an image. Our weight layer configuration is: conv64-conv256-conv256-conv256-conv256-full4096-full4096-full1000, where convN denotes a convolutional layer with N filters, fullM -a fully-connected layer with M outputs. On ILSVRC-2013 validation set, the network achieves the top-1/top-5 classification error of 39.7%/17.7%, which is slightly better than 40.7%/18.2%, reported in [8] for a single ConvNet.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
In this section we describe a technique for visualising the class models, learnt by the image classification ConvNets. Given a learnt classification ConvNet and a class of interest, the visualisation method consists in numerically generating an image [5], which is representative of the class in terms of the ConvNet class scoring model.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
More formally, let S c (I) be the score of the class c, computed by the classification layer of the ConvNet for an image I. We would like to find an L 2 -regularised image, such that the score S c is high:
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
arg max
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
where λ is the regularisation parameter. A locally-optimal I can be found by the back-propagation method. The procedure is related to the ConvNet training procedure, where the back-propagation is used to optimise the layer weights. The difference is that in our case the optimisation is performed with respect to the input image, while the weights are fixed to those found during the training stage.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
We initialised the optimisation with the zero image (in our case, the ConvNet was trained on the zero-centred image data), and then added the training set mean image to the result. The class model visualisations for several classes are shown in Fig. 1.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
It should be noted that we used the (unnormalised) class scores S c , rather than the class posteriors, returned by the soft-max layer: P c = exp Sc c exp Sc . The reason is that the maximisation of the class posterior can be achieved by minimising the scores of other classes. Therefore, we optimise S c to ensure that the optimisation concentrates only on the class in question c. We also experimented with optimising the posterior P c , but the results were not visually prominent, thus confirming our intuition.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
In this section we describe how a classification ConvNet can be queried about the spatial support of a particular class in a given image. Given an image I 0 , a class c, and a classification ConvNet with the class score function S c (I), we would like to rank the pixels of I 0 based on their influence on the score S c (I 0 ).
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
We start with a motivational example. Consider the linear score model for the class c:
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
where the image I is represented in the vectorised (one-dimensional) form, and w c and b c are respectively the weight vector and the bias of the model. In this case, it is easy to see that the magnitude of elements of w defines the importance of the corresponding pixels of I for the class c.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
In the case of deep ConvNets, the class score S c (I) is a highly non-linear function of I, so the reasoning of the previous paragraph can not be immediately applied. However, given an image I 0 , we can approximate S c (I) with a linear function in the neighbourhood of I 0 by computing the first-order Taylor expansion:
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
where w is the derivative of S c with respect to the image I at the point (image) I 0 :
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Another interpretation of computing the image-specific class saliency using the class score derivative (4) is that the magnitude of the derivative indicates which pixels need to be changed the least to affect the class score the most. One can expect that such pixels correspond to the object location in the image. We note that a similar technique has been previously applied by [1] in the context of Bayesian classification.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Given an image I 0 (with m rows and n columns) and a class c, the class saliency map M ∈ R m×n is computed as follows. First, the derivative w (4) is found by back-propagation. After that, the saliency map is obtained by rearranging the elements of the vector w. In the case of a grey-scale image, the number of elements in w is equal to the number of pixels in I 0 , so the map can be computed as M ij = |w h(i,j) |, where h(i, j) is the index of the element of w, corresponding to the image pixel in the i-th row and j-th column. In the case of the multi-channel (e.g. RGB) image, let us assume that the colour channel c of the pixel (i, j) of image I corresponds to the element of w with the index h(i, j, c). To derive a single class saliency value for each pixel (i, j), we took the maximum magnitude of w across all colour channels: M ij = max c |w h(i,j,c) |.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
It is important to note that the saliency maps are extracted using a classification ConvNet trained on the image labels, so no additional annotation is required (such as object bounding boxes or segmentation masks). The computation of the image-specific saliency map for a single class is extremely quick, since it only requires a single back-propagation pass.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
We visualise the saliency maps for the highest-scoring class (top-1 class prediction) on randomly selected ILSVRC-2013 test set images in Fig. 2. Similarly to the ConvNet classification procedure [8], where the class predictions are computed on 10 cropped and reflected sub-images, we computed 10 saliency maps on the 10 sub-images, and then averaged them.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
The weakly supervised class saliency maps (Sect. 3.1) encode the location of the object of the given class in the given image, and thus can be used for object localisation (in spite of being trained on image labels only). Here we briefly describe a simple object localisation procedure, which we used for the localisation task of the ILSVRC-2013 challenge [12].
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Given an image and the corresponding class saliency map, we compute the object segmentation mask using the GraphCut colour segmentation [3]. The use of the colour segmentation is motivated by the fact that the saliency map might capture only the most discriminative part of an object, so saliency thresholding might not be able to highlight the whole object. Therefore, it is important to be able to propagate the thresholded map to other parts of the object, which we aim to achieve here using the colour continuity cues. Foreground and background colour models were set to be the Gaussian Mixture Models. The foreground model was estimated from the pixels with the saliency higher than a threshold, set to the 95% quantile of the saliency distribution in the image; the background model was estimated from the pixels with the saliency smaller than the 30% quantile (Fig. 3, right-middle). The GraphCut segmentation [3] was then performed using the publicly available implementation 2 .
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Once the image pixel labelling into foreground and background is computed, the object segmentation mask is set to the largest connected component of the foreground pixels (Fig. 3, right).
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
We entered our object localisation method into the ILSVRC-2013 localisation challenge. Considering that the challenge requires the object bounding boxes to be reported, we computed them as the bounding boxes of the object segmentation masks. The procedure was repeated for each of the top-5 predicted classes. The method achieved 46.4% top-5 error on the test set of ILSVRC-2013. It should be noted that the method is weakly supervised (unlike the challenge winner with 29.9% error), and the object localisation task was not taken into account during training. In spite of its simplicity, the method still outperformed our submission to ILSVRC-2012 challenge (which used the same dataset), which achieved 50.0% localisation error using a fully-supervised algorithm based on the part-based models [6] and Fisher vector feature encoding [11].
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
In this section we establish the connection between the gradient-based visualisation and the DeconvNet architecture of [13]. As we show below, DeconvNet-based reconstruction of the n-th layer input X n is either equivalent or similar to computing the gradient of the visualised neuron ac-  tivity f with respect to X n , so DeconvNet effectively corresponds to the gradient back-propagation through a ConvNet.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
For the convolutional layer X n+1 = X n K n , the gradient is computed as ∂f /∂X n = ∂f /∂X n+1 K n , where K n and K n are the convolution kernel and its flipped version, respectively. The convolution with the flipped kernel exactly corresponds to computing the n-th layer reconstruction R n in a DeconvNet: R n = R n+1 K n .
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
For the RELU rectification layer X n+1 = max(X n , 0), the sub-gradient takes the form: ∂f /∂X n = ∂f /∂X n+1 1 (X n > 0), where 1 is the element-wise indicator function. This is slightly different from the DeconvNet RELU reconstruction: R n = R n+1 1 (R n+1 > 0), where the sign indicator is computed on the output reconstruction R n+1 instead of the layer input X n .
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Finally, consider a max-pooling layer X n+1 (p) = max q∈Ω(p) X n (q), where the element p of the output feature map is computed by pooling over the corresponding spatial neighbourhood Ω(p) of the input. The sub-gradient is computed as ∂f /∂X n (s) = ∂f /∂X n+1 (p) 1(s = arg max q∈Ω(p) X n (q)). Here, arg max corresponds to the max-pooling "switch" in a DeconvNet.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
We can conclude that apart from the RELU layer, computing the approximate feature map reconstruction R n using a DeconvNet is equivalent to computing the derivative ∂f /∂X n using backpropagation, which is a part of our visualisation algorithms. Thus, gradient-based visualisation can be seen as the generalisation of that of [13], since the gradient-based techniques can be applied to the visualisation of activities in any layer, not just a convolutional one. In particular, in this paper we visualised the class score neurons in the final fully-connected layer.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
It should be noted that our class model visualisation (Sect. 2) depicts the notion of a class, memorised by a ConvNet, and is not specific to any particular image. At the same time, the class saliency visualisation (Sect. 3) is image-specific, and in this sense is related to the image-specific convolutional layer visualisation of [13] (the main difference being that we visualise a neuron in a fully connected layer rather than a convolutional layer).
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
In this paper, we presented two visualisation techniques for deep classification ConvNets. The first generates an artificial image, which is representative of a class of interest. The second computes an image-specific class saliency map, highlighting the areas of the given image, discriminative with respect to the given class. We showed that such saliency map can be used to initialise GraphCutbased object segmentation without the need to train dedicated segmentation or detection models. Finally, we demonstrated that gradient-based visualisation techniques generalise the DeconvNet reconstruction procedure [13]. In our future research, we are planning to incorporate the imagespecific saliency maps into learning formulations in a more principled manner.
SmoothGrad: removing noise by adding noise
Interpreting complex machine learning models, such as deep neural networks, remains a challenge. Yet an understanding of how such models function is important both for building applications and as a problem in its own right. From health care domains (Hughes et al., 2016;Doshi-Velez et al., 2014;Lou et al., 2012) to education (Kim et al., 2015), there are many domains where interpretability is important. For example, the pneumonia risk prediction case study in (Lou et al., 2012) showed that more interpretable models can reveal important but surprising patterns in the data that complex models overlooked. For reviews of interpretable models, see (Freitas, 2014;Doshi-Velez, 2017).
SmoothGrad: removing noise by adding noise
Interpreting complex machine learning models, such as deep neural networks, remains a challenge. Yet an understanding of how such models function is important both for building applications and as a problem in its own right. From health care domains (Hughes et al., 2016;Doshi-Velez et al., 2014;Lou et al., 2012) to education (Kim et al., 2015), there are many domains where interpretability is important. For example, the pneumonia risk prediction case study in (Lou et al., 2012) showed that more interpretable models can reveal important but surprising patterns in the data that complex models overlooked. For reviews of interpretable models, see (Freitas, 2014;Doshi-Velez, 2017).
SmoothGrad: removing noise by adding noise
One case of interest is image classification systems. Finding an "explanation" for a classification decision could potentially shed light on the underlying mechanisms of such systems, as well as helping in enhancing them. For example, the technique of deconvolution helped researchers identify neurons that failed to learn any meaningful features, knowledge that was used to improve the network, as in (Zeiler & Fergus, 2014).
SmoothGrad: removing noise by adding noise
A common approach to understanding the decisions of image classification systems is to find regions of an image that were particularly influential to the final classification. (Baehrens et al., 2010;Zeiler & Fergus, 2014;Springenberg et al., 2014;Zhou et al., 2016;Selvaraju et al., 2016;Sundararajan et al., 2017;Zintgraf et al., 2016). These approaches (variously called sensitivity maps, saliency maps, or pixel attribution maps; see discussion in Section 2; use occlusion techniques or calculations with gradients to assign an "importance" value to individual pixels which is meant to reflect their influence on the final classification.
SmoothGrad: removing noise by adding noise
A common approach to understanding the decisions of image classification systems is to find regions of an image that were particularly influential to the final classification. (Baehrens et al., 2010;Zeiler & Fergus, 2014;Springenberg et al., 2014;Zhou et al., 2016;Selvaraju et al., 2016;Sundararajan et al., 2017;Zintgraf et al., 2016). These approaches (variously called sensitivity maps, saliency maps, or pixel attribution maps; see discussion in Section 2; use occlusion techniques or calculations with gradients to assign an "importance" value to individual pixels which is meant to reflect their influence on the final classification.
SmoothGrad: removing noise by adding noise
In practice these techniques often do seem to highlight regions that can be meaningful to humans, such as the eyes in a face. At the same time, sensitivity maps are often visually noisy, highlighting some pixels that-to a human eye-seem randomly selected. Of course, a priori we cannot determine if this noise reflects an underlying truth about how networks perform classification, or is due to more superficial factors. Either way, it seems like a phenomenon worth investigating further. This paper describes a very simple technique, SMOOTH-GRAD, that in practice tends to reduce visual noise, and also can be combined with other sensitivity map algorithms. The core idea is to take an image of interest, sample similar images by adding noise to the image, then take the average of the resulting sensitivity maps for each sampled image. We also find that the common regularization technique of adding noise at training time (Bishop, 1995) has an additional "de-noising" effect on sensitivity maps. The two techniques (training with noise, and inferring with noise) seem to have additive effect; performing them together yields the best results. This paper compares the SMOOTHGRAD method to several gradient-based sensitivity map methods and demonstrates its effects. We provide a conjecture, backed by some empirical evidence, for why the technique works, and why it might be more reflective of how the network is doing classification. We also discuss several ways to enhance visualizations of these sensitivity maps. Finally, we also make the code used to generate all the figures in this paper available, along with 200+ examples of each compared method on the web at https://goo.gl/EfVzEE. arXiv:1706.03825v1 [cs.
SmoothGrad: removing noise by adding noise
In practice these techniques often do seem to highlight regions that can be meaningful to humans, such as the eyes in a face. At the same time, sensitivity maps are often visually noisy, highlighting some pixels that-to a human eye-seem randomly selected. Of course, a priori we cannot determine if this noise reflects an underlying truth about how networks perform classification, or is due to more superficial factors. Either way, it seems like a phenomenon worth investigating further. This paper describes a very simple technique, SMOOTH-GRAD, that in practice tends to reduce visual noise, and also can be combined with other sensitivity map algorithms. The core idea is to take an image of interest, sample similar images by adding noise to the image, then take the average of the resulting sensitivity maps for each sampled image. We also find that the common regularization technique of adding noise at training time (Bishop, 1995) has an additional "de-noising" effect on sensitivity maps. The two techniques (training with noise, and inferring with noise) seem to have additive effect; performing them together yields the best results. This paper compares the SMOOTHGRAD method to several gradient-based sensitivity map methods and demonstrates its effects. We provide a conjecture, backed by some empirical evidence, for why the technique works, and why it might be more reflective of how the network is doing classification. We also discuss several ways to enhance visualizations of these sensitivity maps. Finally, we also make the code used to generate all the figures in this paper available, along with 200+ examples of each compared method on the web at https://goo.gl/EfVzEE. arXiv:1706.03825v1 [cs.
SmoothGrad: removing noise by adding noise
LG] 12 Jun 2017 2. Gradients as sensitivity maps Consider a system that classifies an image into one class from a set C. Given an input image x, many image classification networks (Szegedy et al., 2016;LeCun et al., 1998) compute a class activation function S c for each class c ∈ C, and the final classification class(x) is determined by which class has the highest score. That is,
SmoothGrad: removing noise by adding noise
LG] 12 Jun 2017 2. Gradients as sensitivity maps Consider a system that classifies an image into one class from a set C. Given an input image x, many image classification networks (Szegedy et al., 2016;LeCun et al., 1998) compute a class activation function S c for each class c ∈ C, and the final classification class(x) is determined by which class has the highest score. That is,
SmoothGrad: removing noise by adding noise
A mathematically clean way of locating "important" pixels in the input image has been proposed by several authors, e.g., (Baehrens et al., 2010;Simonyan et al., 2013;Erhan et al., 2009). If the functions S c are piecewise differentiable, for any image x one can construct a sensitivity map M c (x) simply by differentiating M c with respect to the input, x. In particular, we can define
SmoothGrad: removing noise by adding noise
A mathematically clean way of locating "important" pixels in the input image has been proposed by several authors, e.g., (Baehrens et al., 2010;Simonyan et al., 2013;Erhan et al., 2009). If the functions S c are piecewise differentiable, for any image x one can construct a sensitivity map M c (x) simply by differentiating M c with respect to the input, x. In particular, we can define
SmoothGrad: removing noise by adding noise
Here ∂S c represents the derivative (i.e. gradient) of S c . Intuitively speaking, M c represents how much difference a tiny change in each pixel of x would make to the classification score for class c. As a result, one might hope that the resulting map M c would highlight key regions.
SmoothGrad: removing noise by adding noise
In practice, the sensitivity map of a label does seem to show a correlation with regions where that label is present (Baehrens et al., 2010;Simonyan et al., 2013). However, the sensitivity maps based on raw gradients are typically visually noisy, as shown in Fig. 1. Moreover, as this image shows, the correlations with regions a human would pick out as meaningful are rough at best. 
SmoothGrad: removing noise by adding noise
In practice, the sensitivity map of a label does seem to show a correlation with regions where that label is present (Baehrens et al., 2010;Simonyan et al., 2013). However, the sensitivity maps based on raw gradients are typically visually noisy, as shown in Fig. 1. Moreover, as this image shows, the correlations with regions a human would pick out as meaningful are rough at best. 
SmoothGrad: removing noise by adding noise
There are several hypotheses for the apparent noise in raw gradient visualizations. One possibility, of course, is that the maps are faithful descriptions of what the network is doing. Perhaps certain pixels scattered, seemingly at random, across the image are central to how the network is making a decision. On the other hand, it is also possible that using the raw gradient as a proxy for feature importance is not optimal. Seeking better explanations of network decisions, several prior works have proposed modifications to the basic technique of gradient sensitivity maps; we summarize a few key examples here.
SmoothGrad: removing noise by adding noise
One issue with using the gradient as a measure of influence is that an important feature may "saturate" the function S c . In other words, it may have a strong effect globally, but with a small derivative locally. Several approaches, Layerwise Relevance Propagation (Bach et al., 2015), DeepLift (Shrikumar et al., 2017), and more recently Integrated Gradients (Sundararajan et al., 2017), attempt to address this potential problem by estimating the global importance of each pixel, rather than local sensitivity. Maps created with these techniques are referred to as "saliency" or "pixel attribution" maps.
SmoothGrad: removing noise by adding noise
One issue with using the gradient as a measure of influence is that an important feature may "saturate" the function S c . In other words, it may have a strong effect globally, but with a small derivative locally. Several approaches, Layerwise Relevance Propagation (Bach et al., 2015), DeepLift (Shrikumar et al., 2017), and more recently Integrated Gradients (Sundararajan et al., 2017), attempt to address this potential problem by estimating the global importance of each pixel, rather than local sensitivity. Maps created with these techniques are referred to as "saliency" or "pixel attribution" maps.
SmoothGrad: removing noise by adding noise
Another strategy for enhancing sensitivity maps has been to change or extend the backpropagation algorithm itself, with the goal of emphasizing positive contributions to the final outcome. Two examples are the Deconvolution (Zeiler & Fergus, 2014) and Guided Backpropagation (Springenberg et al., 2014) techniques, which modify the gradients of ReLU functions by discarding negative values during the backpropagation calculation. The intention is to perform a type of "deconvolution" which will more clearly show features that triggered activations of high-level units. Similar ideas appear in (Selvaraju et al., 2016;Zhou et al., 2016), which suggest ways to combine gradients of units at multiple levels.
SmoothGrad: removing noise by adding noise
Another strategy for enhancing sensitivity maps has been to change or extend the backpropagation algorithm itself, with the goal of emphasizing positive contributions to the final outcome. Two examples are the Deconvolution (Zeiler & Fergus, 2014) and Guided Backpropagation (Springenberg et al., 2014) techniques, which modify the gradients of ReLU functions by discarding negative values during the backpropagation calculation. The intention is to perform a type of "deconvolution" which will more clearly show features that triggered activations of high-level units. Similar ideas appear in (Selvaraju et al., 2016;Zhou et al., 2016), which suggest ways to combine gradients of units at multiple levels.
SmoothGrad: removing noise by adding noise
In what follows, we provide detailed comparisons of "vanilla" gradient maps with those created by integrated gradient methods and guided backpropagation. A note on terminology: although the terms "sensitivity map", "saliency map", and "pixel attribution map" have been used in different contexts, in this paper, we will refer to these methods collectively as "sensitivity maps."
SmoothGrad: removing noise by adding noise
There is a possible explanation for the noise in sensitivity maps, which to our knowledge has not been directly addressed in the literature: the derivative of the function S c may fluctuate sharply at small scales. In other words, the apparent noise one sees in a sensitivity map may be due to essentially meaningless local variations in partial derivatives. After all, given typical training techniques there is no reason to expect derivatives to vary smoothly. Indeed, the networks in question typically are based on ReLU activation functions, so S c generally will not even be continu-ously differentiable. Fig. 2 gives example of strongly fluctuating partial derivatives. This fixes a particular image x, and an image pixel x i , and plots the values of ∂Sc ∂xi (t) as fraction of the maximum entry in the gradient vector, max i ∂Sc ∂xi (t), for a short line segment x + t in the space of images parameterized by t ∈ [0, 1]. We show it as a fraction of the maximum entry in order to verify that the fluctuations are significant. The length of this segment is small enough that the starting image x and the final image x + looks the same to a human. Furthermore, each image along the path is correctly classified by the model. The partial derivatives with respect to the red, green, and blue components, however, change significantly. Figure 2. The partial derivative of Sc with respect to the RGB values of a single pixel as a fraction of the maximum entry in the gradient vector, maxi ∂Sc ∂x i (t), (middle plot) as one slowly moves away from a baseline image x (left plot) to a fixed location x + (right plot). is one random sample from N (0, 0.01 2 ). The final image (x + ) is indistinguishable to a human from the origin image x.
SmoothGrad: removing noise by adding noise
Given these rapid fluctuations, the gradient of S c at any given point will be less meaningful than a local average of gradient values. This suggests a new way to create improved sensitivity maps: instead of basing a visualization directly on the gradient ∂S c , we could base it on a smoothing of ∂S c with a Gaussian kernel.
SmoothGrad: removing noise by adding noise
Directly computing such a local average in a highdimensional input space is intractable, but we can compute a simple stochastic approximation. In particular, we can take random samples in a neighborhood of an input x, and average the resulting sensitivity maps. Mathematically, this means calculatinĝ
SmoothGrad: removing noise by adding noise
where n is the number of samples, and N (0, σ 2 ) represents Gaussian noise with standard deviation σ. We refer to this method as SMOOTHGRAD throughout the paper.
SmoothGrad: removing noise by adding noise
To assess the SMOOTHGRAD technique, we performed a series of experiments using a neural network for image classification (Szegedy et al., 2016;TensorFlow, 2017). The results suggest the estimated smoothed gradient,M c , leads to visually more coherent sensitivity maps than the unsmoothed gradient M c , with the resulting visualizations aligning better-to the human eye-with meaningful features.
SmoothGrad: removing noise by adding noise
To assess the SMOOTHGRAD technique, we performed a series of experiments using a neural network for image classification (Szegedy et al., 2016;TensorFlow, 2017). The results suggest the estimated smoothed gradient,M c , leads to visually more coherent sensitivity maps than the unsmoothed gradient M c , with the resulting visualizations aligning better-to the human eye-with meaningful features.
SmoothGrad: removing noise by adding noise
Our experiments were carried out using an Inception v3 model (Szegedy et al., 2016) that was trained on the ILSVRC-2013 dataset (Russakovsky et al., 2015) and a convolutional MNIST model based on the TensorFlow tutorial (TensorFlow, 2017).
SmoothGrad: removing noise by adding noise
Our experiments were carried out using an Inception v3 model (Szegedy et al., 2016) that was trained on the ILSVRC-2013 dataset (Russakovsky et al., 2015) and a convolutional MNIST model based on the TensorFlow tutorial (TensorFlow, 2017).
SmoothGrad: removing noise by adding noise
Sensitivity maps are typically visualized as heatmaps. Finding the right mapping from a channel values at a pixel to a particular color turns out to be surprisingly nuanced, and can have a large effect on the resulting impression of the visualization. This section summarizes some visualization techniques and lessons learned in the process of comparing various sensitivity map work. Some of these techniques may be universally useful regardless of the choice of sensitivity map methods.
SmoothGrad: removing noise by adding noise
Sensitivity map algorithms often produce signed values. There is considerable ambiguity in how to convert signed values to colors. A key choice is whether to represent positive and negative values differently, or to visualize the absolute value only. The utility of taking the absolute values of gradients or not depends on the characteristics of the dataset of interest. For example, when the object of interest has the same color across the classes (e.g., digits are always white in MNIST digits (LeCun et al., 2010)), the positive gradients indicate positive signal to the class. On the other hand, for ImageNet dataset (Russakovsky et al., 2015), we have found that taking the absolute value of the gradient produced clearer pictures. One possible explanation for this phenomenon is that the direction is context dependent: many image recognition tasks are invariant under color and illumination changes. For instance, in classifying a ball, a dark ball on a bright background would have negative gradient, while white ball on darker background would have a positive gradient.
SmoothGrad: removing noise by adding noise
Another property of the gradient that we observe is the presence of few pixels that have much higher gradients than the average. This is not a new discovery -this property was utilized in generating adversarial examples that are in- Figure 3. Effect of noise level (columns) on our method for 5 images of the gazelle class in ImageNet (rows). Each sensitivity map is obtained by applying Gaussian noise N (0, σ 2 ) to the input pixels for 50 samples, and averaging them. The noise level corresponds to σ/(xmax − xmin).
SmoothGrad: removing noise by adding noise
distinguishable to humans (Szegedy et al., 2013). These outlying values have the potential to throw off color scales completely. Capping those extreme values to a relatively high value (we find 99 th percentile to be sufficient) leads to more visually coherent maps as in (Sundararajan et al., 2017). Without this post-processing step, maps may end up almost entirely black.
SmoothGrad: removing noise by adding noise
Some techniques create a final sensitivity map by multiplying gradient-based values and actual pixel values (Shrikumar et al., 2017;Sundararajan et al., 2017). This multiplication does tend to produce visually simpler and sharper images, although it can be unclear how much of this can be attributed to sharpness in the original image itself. For example, a black/white edge in the input can lead to an edge-like structure on the final visualization even if the underlying sensitivity map has no edges.
SmoothGrad: removing noise by adding noise
However, this may result in undesired side effect. Pixels with values of 0 will never show up on the sensitivity map. For example, if we encode black as 0, the image of a classifier that correctly predicts a black ball on a white background will never highlight the black ball in the image.
SmoothGrad: removing noise by adding noise
On the other hand, multiplying gradients with the input images makes sense when we view the importance of the feature as their contribution to the total score, y. For example, in a linear system y = W x, it makes sense to consider x i w i as the contribution of x i to the final score y.
SmoothGrad: removing noise by adding noise
For these reasons, we show our results with and without the image multiplication in Fig. 5.
SmoothGrad: removing noise by adding noise
SMOOTHGRAD has two hyper-parameters: σ, the noise level or standard deviation of the Gaussian perturbations, and n, the number of samples to average over.
SmoothGrad: removing noise by adding noise
Noise, σ Fig. 3 shows the effect of noise level for several example images from ImageNet (Russakovsky et al., 2015). The 2 nd column corresponds to the standard gradient (0% noise), which we will refer to as the "Vanilla" method throughout the paper. Since quantitative evaluation of a map remains an unsolved problem, we again focus on qualitative evaluation. We observe that applying 10%-20% noise (middle columns) seems to balance the sharpness of sensitivity map and maintain the structure of the original image.We also observe that while this range of noise gives generally good results for Inception, the ideal noise level depends on the input. See Fig. 10 for a similar experiment on the MNIST dataset.
SmoothGrad: removing noise by adding noise
Sample size, n
SmoothGrad: removing noise by adding noise
In Fig. 4 we show the effect of sample size, n. As expected, the estimated gradient becomes smoother as the sample size, n, increases. We empirically found a diminishing return -there was little apparent change in the visualizations for n > 50.
SmoothGrad: removing noise by adding noise
Since there is no ground truth to allow for quantitative evaluation of sensitivity maps, we follow prior work (Simonyan et al., 2013;Zeiler & Fergus, 2014;Springenberg et al., 2014;Selvaraju et al., 2016;Sundararajan et al., 2017) and focus on two aspects of qualitative evaluation.
SmoothGrad: removing noise by adding noise
Since there is no ground truth to allow for quantitative evaluation of sensitivity maps, we follow prior work (Simonyan et al., 2013;Zeiler & Fergus, 2014;Springenberg et al., 2014;Selvaraju et al., 2016;Sundararajan et al., 2017) and focus on two aspects of qualitative evaluation.
SmoothGrad: removing noise by adding noise
First, we inspect visual coherence (e.g., the highlights are only on the object of interest, not the background). Second, we test for discriminativity, where in an image with both a monkey and a spoon, one would expect an explanation for a monkey classification to be concentrated on the monkey rather than the spoon, and vice versa.
SmoothGrad: removing noise by adding noise
Regarding visual coherence, Fig. 5 shows a side-by-side comparison between our method and three gradient-based methods: Integrated Gradients (Sundararajan et al., 2017), Guided BackProp (Springenberg et al., 2014) and vanilla gradient. Among a random sample of 200 images that we inspected, we found SMOOTHGRAD to consistently provide more visually coherent maps than Integrated Gradients and vanilla gradient. While Guided BackProp provides the most sharp maps (last three rows of Fig. 5), it is prone to failure (first three rows of Fig. 5), especially for images with uniform background. On the contrary, our observation is that SMOOTHGRAD has the highest impact when the object is surrounded with uniform background color (first three rows of Fig. 5). Exploring this difference is an interesting area for investigation. It is possible that the smoothness of the class score function may be related to spatial statistics of the underlying image; noise may have a differential effect on the sensitivity to different textures. For these images, SMOOTHGRAD qualitatively shows better discriminativity over the other methods.
SmoothGrad: removing noise by adding noise
It remains an open question to understand which properties affect the discriminativity of a given method -e.g. understanding why Guided BackProp seems to show the weakest discriminativity.
SmoothGrad: removing noise by adding noise
One can think of SMOOTHGRAD as smoothing the vanilla gradient method using a simple procedure: averaging the vanilla sensitivity maps of n noisy images. With that in mind, the same smoothing procedure can be used to augment any gradient-based method. In Fig. 7 we show the results of applying SMOOTHGRAD in combination with Integrated Gradients and Guided BackProp. We observe that this augmentation improves the visual coherence of sensitivity maps for both methods.
SmoothGrad: removing noise by adding noise
For further analysis, we point the reader to our web page at https://goo.gl/EfVzEE with sensitivity maps of 200+ images and four different methods.
SmoothGrad: removing noise by adding noise
SMOOTHGRAD as discussed so far may be applied to classification networks as-is. In situations where there is a premium on legibility, however, it is natural to ask whether there is a similar way to modify the network weights so that its sensitivity maps are sharper. One idea that is parallel in some ways to SMOOTHGRAD is the well-known regularization technique of adding noise to samples during training (Bishop, 1995). We find that the same method also improves the sharpness of the sensitivity map. Fig. 8 and Fig. 9 show the effect of adding noise at training time and/or evaluation time for the MNIST and Inception model respectively. Interestingly, adding noise at training time seems to also provide a de-noising effect on the sensitivity map. Lastly, the two techniques (training with noise, and inferring with noise) seem to have additive effect; performing them together produces the most visually coherent map of the 4 combinations.
SmoothGrad: removing noise by adding noise
SMOOTHGRAD as discussed so far may be applied to classification networks as-is. In situations where there is a premium on legibility, however, it is natural to ask whether there is a similar way to modify the network weights so that its sensitivity maps are sharper. One idea that is parallel in some ways to SMOOTHGRAD is the well-known regularization technique of adding noise to samples during training (Bishop, 1995). We find that the same method also improves the sharpness of the sensitivity map. Fig. 8 and Fig. 9 show the effect of adding noise at training time and/or evaluation time for the MNIST and Inception model respectively. Interestingly, adding noise at training time seems to also provide a de-noising effect on the sensitivity map. Lastly, the two techniques (training with noise, and inferring with noise) seem to have additive effect; performing them together produces the most visually coherent map of the 4 combinations.
SmoothGrad: removing noise by adding noise
The experiments described here suggest that gradientbased sensitivity maps can be sharpened by two forms of smoothing. First, averaging maps made from many small perturbations of a given image seems to have a significant smoothing effect. Second, that effect can be enhanced further by training on data that has been perturbed with random noise.
SmoothGrad: removing noise by adding noise
These results suggest several avenues for future research. First, while we have provided a plausibility argument for our conjecture that noisy sensitivity maps are due to noisy gradients, it would be worthwhile to look for further evidence and theoretical arguments that support or disconfirm this hypothesis. It is certainly possible that the sharpening effect of SMOOTHGRAD has other causes, such as a differential effect of random noise on different textures.
SmoothGrad: removing noise by adding noise
Second, in addition to training with noise, there may be more direct methods to create systems with smoother class score functions. For example, one could train with an explicit penalty on the size of partial derivatives. To create      more spatial coherent maps, one could add a penalty for large differences in partial derivatives of the class score with respect to neighboring pixels. It may also be worth investigating the geometry of the class score function to understand why smoothing seems to be more effective on images with large regions of near-constant pixel values.
SmoothGrad: removing noise by adding noise
A further area for exploration is to find better metrics for comparing sensitivity maps. To measure spatial coherence, one might use existing databases of image segmentations, and we are already making progress (Oh et al., 2017;Selvaraju et al., 2016). Systematic measurements of discriminativity could also be valuable. Finally, a natural question is whether the de-noising techniques described here generalize to other network architectures and tasks.