Is the huggingface run_mlm Script dynamically masked?
Hi,In the roberta paper, the model is trained by dynamic masking of sentences.If a roberta model is further pre-trained using therun_mlm.py, is the sentences going to be dynamically masked during pre-training using this file or is it statically masked like vanilla BERT?The script is found heregithub.comtransformers/examples/pytorch/language-modeling at master ·...master/examples/pytorch/language-modeling🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/language-modeling at master · huggingface/transformers
2021-08-21T08:33:33Z
Apparently, dynamic masking is always used, no matter the model. See this:dynamic masking for RoBERTa model · Issue #5979 · huggingface/transformers · GitHub
2021-08-21T09:53:08Z
HelloThats actually coolHave an awesome day Neils!
2021-08-25T12:29:05Z
Hi again,I have one small question,What does the checkpoint folders that are created during the pre training. There folders are eating up my har drive space during training large text files(or when i was pre training line by line).
2022-05-27T06:06:15Z
Hey I don’t quite understand what is being said in the discussion linked by@nielsrCan someone please confirm the claim and explain it?
2022-05-27T16:47:10Z
They are talking about dynamic masking that was mentioned in theRoBERTa paper. Theoriginal BERT paperdid not use dynamic masking, but rather static masking meaning they masked all sentences before training and used those same masks across multiple epochs. Dynamic masking is better because it masks sentences during training, meaning that each epoch would have different masks. Since the masks are different each time, the learning is more challenging and thus converges slower than the original BERT approach.All masked language models (including BertModel) implementations on Huggingface useData Collatorsto mask sentences. Collators are inherently dynamic because they are applied during training. If you want to use static masking then you will have to code that yourself.I hope that clarifies it for you.
2022-05-29T06:12:14Z
Thanks for the explanaton! I still don’t quite get the part where the paper says each sentence is duplicated 10 times and passed into the masking function though. The DataCollator does not do that right?
2022-05-29T15:17:23Z
It is explained in the Roberta paper, section 4.1 static vs dynamic masking. So basically they duplicated their corpus 10 times and masked each sentence in it. That way every sentence gets new masks. They did so over 40 epochs (duplicated corpus 10 * 4), so each sentence with the same masks is trained 4 times.As for the DataCollator class (DataCollatorForLanguageModeling), you have to understand that their code doesn’t follow the original papers. The collator masks a batch of sentences in real-time. So no duplication, but still dynamic masking. The only difference is that the masks might be different each epoch across many epochs. Also, at least in the case of thetorch implementation, masking is done on the batch, and not on each sentence, which means that sometimes sentences might have no masks whatsoever, and some other times sentences might end up with more than 15% masks. Took me a while to figure that out…
2022-06-01T23:46:31Z
Thanks a lot for your reply. Really helped clear things up