Why does Bart decoder's attention mask mark relevant indices with 0 instead of 1?
Hi.When we don’t passdecoder_attention_masktoBartModel, the model automatically creates decoder input masks with_make_causal_mask.I’ve noticed that the method inserts ‘0’ in mask positions corresponding to indices the model needs to attend, and-infin positions corresponding to indices to be ignored. Below is the link to aforementioned code:github.comhuggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/src/transformers/models/bart/modeling_bart.py#L85return shifted_input_idsdef _make_causal_mask(input_ids_shape: torch.Size, dtype: torch.dtype, past_key_values_length: int = 0):"""Make causal mask used for bi-directional self-attention."""bsz, tgt_len = input_ids_shapemask = torch.full((tgt_len, tgt_len), float("-inf"))mask_cond = torch.arange(mask.size(-1))mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)mask = mask.to(dtype)if past_key_values_length > 0:mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype), mask], dim=-1)return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):"""Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.As far as I know attention masks should have 1 in indices we want to attend. Could anyone shed some light on this?
2021-05-31T03:03:46Z
Further investigation shows this behavior is desired since attention mask is added to attention weights, so that 0 attention mask value preserves the inputs while -inf attention mask value “masks out” the inputs. (related code pasted below)However, shouldn’t the encoder attention mask be initialized the same way (0 for relevant inputs, -inf for padding inputs) as well?Currently, the documentation says encoder attention mask values should be 1 for relevant inputs and 0 for padding inputs.github.comhuggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/src/transformers/models/bart/modeling_bart.py#L223if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):raise ValueError(f"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is {attn_weights.size()}")if attention_mask is not None:if attention_mask.size() != (bsz, 1, tgt_len, src_len):raise ValueError(f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}")attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_maskattn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)attn_weights = F.softmax(attn_weights, dim=-1)if layer_head_mask is not None:if layer_head_mask.size() != (self.num_heads,):raise ValueError(f"Head mask for a single layer should be of size {(self.num_heads,)}, but is {layer_head_mask.size()}")attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)