Doubts about attention masks
Hi. My issue relates tothis issue. I wanted to have a graph-like input and was trying to mimic the behaviour of graph neural networks by masking the tokens (nodes) outside of the neighbourhood. Specifically, if I had for example a graph likeI -> am -> hungry, and the mapping for tokens was one to one, I would like to have an attention mask like: [[1, 1, 0], [0, 1, 1], [0, 0, 1]], meaning the tokenIwould attend to itself and the tokenam, but nothungrysince it is outside its neighbourhood.Is this behaviour possible? I have been searching and found theget_extended_attention_maskfunction. It does work to send a 3D attention mask to the model. However,this linesays that the 3D attention should be (batch, from_seq_len, to_seq_len) which suggests that this mask is for cross attention.Areattention_masksfor self-attention or cross-attention?
2024-06-29T20:34:07Z
HuggingFace is still working on it. There is a PR to support passing custom attention masks.Allow passing 2D attention mask · Issue #27640 · huggingface/transformers · GitHub