Why does RoBERTa behave differently if I provide a corpus that contains special tokens?
Hello all,Recently I’ve been working on training from scratch a RoBERTa model starting from the code inthis tutorial.I am working with a specific corpus I prepared according to my own format<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>and I noticed that one of the parameters that can be passed to thetokenizer.encoder_plusfunction isadd_special_tokens.Ifadd_special_tokens=True, the result of the encoding of the sentence<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>becomes<s> <s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s> </s>and thespecial_tokens_maskis 1 0 0 … 0 1When I triedadd_special_tokens=Falseon the same sentence<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>the result of the encoding was correct:<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>However, thespecial_tokens_maskremains 0 0 … 0 0.After testing both versions, the result I got from the first was very good, while the second failed.This raises a few issues that I wasn’t able to solve on my own:How can I access thespecial_tokens_maskto correct it to what it should be?Where does RoBERTa make use of that mask, if it does?Is there a method for setting the mask to something I want? e.g. the mask for<s> ID 10 <i> COUNTRY  USA </s>should be1 0 0 1 0 0 1if<s>,</s>and<i>should all be treated as special tokens.If RoBERTa is not the correct model to do this, what model should I go for?Thanks a lot!
2020-09-11T19:43:14Z
This actually feels more like a bug than a problem on your end. I suspect that the tokenization between the two is identical (i.e.<s>gets the same ID in both cases), but that means that the special_tokens_maskshouldalso be the same. Best to wait for others who might be more certain.
2020-09-14T11:21:32Z
Hello! Thanks for the reply. I checked what you mentioned and yes, you were right.I tried to encode the sentence at the top settingadd_special_tokens=FalseandTrue.Token 4 is<s>, which is both the<cls>token and the<bos>token, while token 6 is</s>, which acts assep_tokenandeos_token.The first case hasadd_special_tokens=Falseand its special token mask is full of 0’s, the first case hasadd_special_tokens=Trueand as expected the<bos>and<eos>tokens were added by the algorithm. The special tokens mask only shows the first and last tokens as “1”, while all the other 4’s and 6’s are missing.<s> ID 10 </s><s> NAME Trevor </s> <s> COUNTRY USA </s><s> CAPITAL Washington DC </s>
[4, 0, 232, 28, 27, 6, 4, 1, 232, 63, 93, 80, 97, 90, 93, 6, 4, 2, 232, 64, 62, 44, 6, 4, 3, 232, 66, 76, 94, 83, 84, 89, 82, 95, 90, 89, 232, 47, 46, 6]

[4, 4, 0, 232, 28, 27, 6, 4, 1, 232, 63, 93, 80, 97, 90, 93, 6, 4, 2, 232, 64, 62, 44, 6, 4, 3, 232, 66, 76, 94, 83, 84, 89, 82, 95, 90, 89, 232, 47, 46, 6, 6]
[1, 0, 0,   0,  0,  0, 0, 0, 0,   0,  0,  0,  0,  0,  0,  0, 0, 0, 0,   0,  0,  0,  0, 0, 0, 0,   0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,   0,  0,  0, 0, 1]
2020-09-18T12:51:19Z
I asked this question in anissueon the main repository, and received an in-depth answer there.
2020-09-23T07:38:10Z
cc@mfuntowiczThis feels like a bug. Special characters do not seem to be encoded correctly in the RoBERTaTokenizer.