BertForMaskedLM train
I have a questionWhen training using BertForMaskedLM, is the train data as below correct?token2idx<pad> : 0, <mask>: 1, <cls>:2, <sep>:3max len : 8input token<cls> hello i <mask> cats <sep>input ids[2, 34,45,1,56,3,0,0]attention_mask[1,1,1,1,1,1,0,0]labels[-100,-100,-100,64,-100,-100,-100,-100]I wonder if I should also assign -100 to labels for padding token.
2021-01-19T10:38:05Z
Hi,Were you able to figure it out? I’m also trying to do the same thing.Thanks,Ayala
2021-01-20T08:05:18Z
you should replace all tokens (including paddding) inlabelswith -100 except the masked tokens so the loss will only be calculated for masked tokens.