Will masking more tokens speed up training and use less memory in HuggingFace's Bert or Roberta?
I am curious how HF implements Bert and Roberta. In theory, dropping more tokens in BertForMaskedLM or RobertaForMaskedLM through attn_mask should speed up training and use less memory since less tokens are attended. For example, if I mask 50%, it should speed up by much more than 2x. Is this the case in HF’s implementations?
2022-12-03T11:29:15Z
Sentences are padded to have the same length.Tokens are masked, not dropped.All tokens are attended (<pad> included) but masked ones get a 0 weight after the softmax computation thanks to the mask.So there is no speedup if sentences have a fixed length even if most tokens are <pad>.