Masking Probability
HiI am wondering whether the masking of tokens [MASK] is done by applying the masking probability to a given sequence or the whole batch altogether.
2020-08-18T12:21:43Z
InDataCollatorForLanguageModelingthe masking is done on the tensor directly.
2020-08-18T12:49:21Z
If I read the code correctly, it is every “token” has masking probability to be masked (makes/replaced/not changed). Independent to how many tokens you have i.e. the size of tensor.
2020-08-19T14:44:12Z
In the paper, It’s mentioned that the masing is done on 0.15 f all WordPiece tokens in each sequence at random.In the code though, it’s based on theinputsinlabels = inputs.clone()
probability_matrix = torch.full(labels.shape, self.mlm_probability)
masked_indices = torch.bernoulli(probability_matrix).bool()I don’t know whether inputs refers to one sequence or one batch of sequences
2020-08-20T15:28:31Z
inputsis batch andprobability_matrixcreates prob for each sequence.