Do automatically generated attention masks ignore padding?
For some reason, I thought that when an attention mask was not given to a model, an attention mask was automatically generated that considers padding tokens. I always include the full encoding of the tokenizer (input IDs, attention mask, type-token) out of habit but then I sawthis examplewhich does not do that. They only pass the input IDs. At first I would have expected this to work fine, but if you include attention masks explicitly you get different results than without.To confirm, I looked at the source code of BERT and I find that, indeed, when an attention mask is not given, a mask is created of batch_size x sequence length of all 1’s (no special treatment for padding).github.comhuggingface/transformers/blob/6ed9882ddb2b6249463c855dcca6860161d91f3e/src/transformers/models/bert/modeling_bert.py#L959-L960if attention_mask is None:attention_mask = torch.ones(((batch_size, seq_length + past_key_values_length)), device=device)So can anyone confirm? Relying on the automatically generated attention mask is not enough because it does not block attention on padding tokens, is that correct? This implies that you should always pass the full encoding of a tokenizer tomodel(**encoding)to ensure that the attention mask is included.
2022-03-08T15:45:16Z
Yes, you need to pass the attention mask returned by the tokenizer. Most models don’t know the padding token ID, so they can’t generate an attention mask that ignores it.
2022-03-08T16:08:00Z
Thanks for confirming! I felt like I was going crazy.Has this ever been different in your memory? I have vague memories where the attention mask was created based on the pad token ID (as it is in the tokenizer currently), but it might be that I am confusing tokenizers source code with model’sforward's.
2022-03-08T16:08:29Z
Not in the past two years at least. Can’t speak for older than this
2022-03-08T16:14:31Z
I went back to 0.6.2 ofpytorch_pretrained_bert, and there it is also simplygithub.comhuggingface/transformers/blob/b832d5bb8a6dfc5965015b828e577677eace601e/pytorch_pretrained_bert/modeling.py#L709-L711def forward(self, input_ids, token_type_ids=None, attention_mask=None, output_all_encoded_layers=True):if attention_mask is None:attention_mask = torch.ones_like(input_ids)So I must have been switching things up. Thanks for giving me some peace of mind.