Attentions not returned from transformers ViT model when using output_attentions=True
I’m usingthis code snippetfrom the docs of HuggingFace ViT classification model - with one addition: I’m using theoutput_attentions=Trueparameter. Nevertheless, no attentions are returned.from transformers import ViTFeatureExtractor, ViTForImageClassification
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = ViTFeatureExtractor.from_pretrained('google/vit-base-patch16-224')
model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224',output_attentions=True)

inputs = feature_extractor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits

# --> this should print the attentions
print(output.attentions)

# model predicts one of the 1000 ImageNet classes
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])The output ofprint(output.attentions)is:attentions=(None, None, None, None, None, None, None, None, None, None, None, None)What am I doing wrong, and how can I get the attentions values?
2024-06-09T20:03:00Z
Hi,You need to pass the flag to the forward of the model:outputs = model(**inputs, output_attentions=True)
2024-06-09T20:27:32Z
Still doesn’t work, but it seems to be related to version 4.41, as mentioned in this Github issue:https://github.com/huggingface/transformers/issues/30978I’m still not sure how to resolve it, though
2024-06-10T00:26:53Z
solved by addingattn_implementation="eager":model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224", attn_implementation="eager")
2024-07-10T19:50:55Z
It works for me.