Pass a custom mask when using RoBERTa
Suppose I have a sequence that consists of 2 sentences separated by </SEP> tokens like A </SEP> B. When performing forward pass with RoBERTa model, I want tokens in sentence A only attend to tokens in sentence A and vice versa for sentence B. The mask will be look like this:In summary, is there any way to explicitly pass a custom attention mask to the model?Thanks in advance.
2021-01-19T15:40:53Z
Attention mask is normally created from input_mask . You cannot bypass attention mask directly. I might be wrong also.For your purpose, create an input_mask with 1s on First row rows and two colums and then 1s on last two rows and last two columns. Set else to 0.
2021-01-19T21:22:45Z
Yes, all models take anattention_maskargument that you can customize.
2021-01-20T02:37:17Z
From the documentation I see that model will accept 1D tensor (assume batch size equal 1) but what I need is 2D attention mask. How can I manage to do that? Thanks for your response.attention_mask(torch.FloatTensorof shape((batch_size, sequence_length)), optional) –Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:1 for tokens that arenot masked,0 for tokens that aremasked.
2021-05-26T08:28:48Z
you can change extended_attention_mask inside Roberta model, in order to make a 3D attention mask for your case.I suggest that we can pass extended_attention_mask as a parameter.
2023-01-10T17:24:18Z
I have a similar issue. Can you provide a minimal example on how to use a 3D attention mask per input sequence?