Difference between setting label index to -100 & setting attention mask to 0
According toDocssetting a token label index to-100makes the model not compute loss on such tokens.And attention mask does thesamethingIs the functionality of both the same or does it differ where one or the other is usedThanks
2021-03-17T18:11:43Z
No the attention mask is not used in the loss computation. It’s just there to make sure your model is not paying attention to the masked tokens.The two things should be used together.
2021-03-17T18:29:37Z
Thanks@sgugger, So If I’m Ignoring the special tokens - CLS/SEP in loss computation using -100, is it recommended to mask them out?
2021-03-17T18:32:02Z
Yes, and you probably don’t want to ignore them with the attention mask (like we do with the padding) since they give useful information to the model.
2021-03-17T18:39:04Z
Alrighty, So in short,special tokens- attend & don’t compute losspadding- don’t attend & don’t compute loss
2021-03-17T19:05:06Z
That would be my recommendation, yes.