Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:

a special mask token with probability 0.8
a random token different from the one masked with probability 0.1
the same token with probability 0.1