Distilbart-mnli-12-9
@valhallaIn distilbart, can i identify the weight of the words in the sequence associated to the candidate label/class. I want to narrow down on the reason for the model assigning a particular score to a given class. For example if “This is awesome anyone thinking to buy should blindly go for it” is assigned a positive label score of 0.99, then is it possible to identify the words in the sequence which carry the most weight/ contribute the most to the positive label, ( in this sequence  the words/phrases - awesome, blindly go for it ) and the relative weight(cosine similarity/distance) of those words to the identified class(i.e positive).Can this be done by accessing or manipulating the end layers of the model or by any other method?Thank you for your help in advance!
2021-01-06T09:08:23Z
@joeddavmight have some ideas here.
2021-01-06T14:37:58Z
You can look at the attention weights. This isnot exactly “explanation”, but alsonot entirelynotexplanationeitherThat is, it can give you some notion of how “important” a word is, but is very limited in its explanatory power. Models like these are difficult to interpret.Passoutput_attentions=Trueto get the attention patterns back when you call the model. You’ll probably want the attention between each word and the EOS token since this embedding is the sequence representation thatBartForSequenceClassificationuses. But which layers and heads are best to use, or how to aggregate their patterns, goes beyond my knowledge.As an aside for any interested readers, we have an awesome tool forvisualizing transformerswhich is really helpful to build intuition about attention patters in transformers.
2021-01-16T15:04:05Z
Hi thank you for your response, ill try to take a deep dive into those that seems like a good place to start.
2021-01-17T14:25:27Z
Hi there is a library released by Pytorch for Model interpretability. It uses the internal embeddings and integrated gradient functions to give the saliency features and attributions of the input words. They have shown its applications for BERT model trained on SQuaD for Question Answering. Is it possible to do the same on these BART models? I am not really well versed with how the pipeline works, is it possible to run it on the distilbart zero-shot models? I am presuming I might have to replicate the zero-shot pipeline and pass the model through their interpretation functions.captum.aiCaptum · Model Interpretability for PyTorchModel Interpretability for PyTorch
2021-01-19T16:50:43Z
I don’t have a complete answer for you, but don’t worry about the pipeline itself as its just a wrapper for the model. I assume this code is designed forBertModel, so the hardest thing you’ll probably need to do is adapt it forBartModelorBartForSequenceClassification.