Regular tokens vs special tokens
Based on the CTRL approach on GPT2, i’m trying to add tokens in order to control my text generation style. Is there a difference between adding a token as a regular one and adding it as a special token?
2021-05-17T10:09:44Z
hey@Felipehonorato, as far as i know, special tokens won’t be split by the tokenizer which might be handy in your case where you’re trying to incorporate control tokens. you can find more information in the docshere.
2021-05-24T10:51:19Z
Really thank you for the reply. Just to make sure i understood:  When adding a word as a regular token, the tokenizer can maybe split this word into subwords, and when passing it as a special token this shouldn’t happen, am I right? I’ve made a couple of tests and by passing as special tokens worked pretty well
2021-05-26T12:20:40Z
yes, you’re totally right regarding the difference between regular vs special tokens! in case you’re interested, this is all being handled by thetokenizerslibrary, which has some extra details here:Input sequences — tokenizers documentation
2023-02-17T17:47:35Z
@lewtunif i have taxonomy of textT1T2T3to establish a context between T1 to T3 , would I need special token or normal tokensT1 [L2] T2 [L3] T3
2024-01-08T05:30:45Z
@lewtunCould you help us in this?Tokenizer.add_tokens automatically convert ESM2 new token to specialWe can not extend the embedding layer by adding new tokens using ESM-2 models.