Cloning dataset in MLM training
Hi there,I wanted to ask if it is a common practice to clone the training dataset when pre-training a model using MLM technique. Because of the probabilities, model will probably always mask different tokens and will have a different task, even when two provided sentences are the same.Example:First iteration of dataset: I ate an apple —masking—> I [MASK] an appleSecond iteration of dataset: I ate an apple --masking–=> I ate [MASK] appleThanks is advance for every comment