Clarification on the attention_mask
Hello everyone,I am trying to build a domain specific RoBERTa model and I need clarification on theattention_maskusage.  Glossary (https://huggingface.co/transformers/glossary.html#attention-mask) says “The attention mask is a binary tensor indicating the position of the padded indices so that the model does not attend to them.”On the github page (https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_roberta.py)lines 196-198:if attention_mask is not None:# Apply the attention mask is (precomputed for all layers in RobertaModel forward() function)attention_scores = attention_scores + attention_maskI would think masking would be a multiplication of attention scores with the attention_mask, not addition. Could someone clarify this for me?My purpose looking into attention_mask is to be able to use it as a quick way of assigning fixed weights to certain tokens.  For example, if I have a domain specific sentence like this: “Parasequence boundariesmay be distinguished bydifferencesinphysicalandchemical properties”, where I care more about the domain specific words than the common language words. I assumed attention_mask could be used as a domain specific weighting. thanks!
2020-10-15T12:47:12Z
Hi! The attention mask is made so that it accepts0s and1s. Putting a1indicates that this token should be attended to, while putting a0indicates a value that should not be attended to.In the models it is modifiedhere(inversion), which means that tokens that have a1value will now have0, and tokens that have a0value will now have-10000.This value is then summed to the attention scoreshere: if tokens had1initially, nothing changes. If tokens has0initially, their attention scores becomes a very negative value, nullifying their impact on the sequence.Unfortunately, you won’t be able to use this attention mask to assign fix weights to certain tokens. Theattention_maskcan be used to nullify the impact of certain tokens on the rest of the sequence, but that’s about it.
2020-10-15T14:03:20Z
Thank you for your response, very clear.  I have missed the inversion part. Now it makes perfect sense. Thanks again!
2023-04-20T16:23:12Z
If you see this, wondering if you pursued this idea@zb1and whether you implemented something of your own or found some other workaround? Thanks!
2024-05-03T16:18:07Z
suppose my sentence is :[‘Mr’ , ‘Obama’, ‘had’, ‘a’, ‘very’, ‘good’, ‘standing’, ‘among’, ‘world’, ‘leaders’ , ‘.’]if I give its attention_mask as :[0, 1,   0, 0, 0, 0, 0, 0, 0, 0 , 0]And since attention, in context of transformers, basically is K.QTthat is key and query ,So my doubt is , what does ignoring mean ??Will the attention score be only attend to Kobama . Qobama ??or does it mean the following :while encoding ‘Obama’ , the self attention will encode it by attending to all other wordsand while encoding other words , the self attention will not look to other words/tokens in the text ??