Why is Code LLama token for prefix, suffix, etc weird underscore character
I noticed these lines of code in the transformers library:github.comhuggingface/transformers/blob/fd6a0ade9b89c415ea213ef1aa07c9b2c32a4d75/src/transformers/models/code_llama/tokenization_code_llama.py#L136-L139prefix_token="▁<PRE>",middle_token="▁<MID>",suffix_token="▁<SUF>",eot_token="▁<EOT>",If you see those funky underscore characters, why are they there?  Isn’t it supposed to be just this in the lines of code:prefix_token="<PRE>",
        middle_token="<MID>",
        suffix_token="<SUF>",
        eot_token="<EOT>",
        fill_token="<FILL_ME>"
2023-10-16T19:36:37Z
@ArthurZdo you know why you put those in?
2023-10-16T19:56:54Z
Hey! That’s because otherwise sentencepiece does not recognize them. And it will split them!The sentencepiece model was trained in such a way
2023-10-16T19:58:33Z
BTW, thanks for replying so quickly!OK then I would not have learned that otherwise from asking you.  Is this an industry known thing that happens with the all of the transformers API, just curious.Also, was it intentional to use that weird underscore character?▁← In your code you are using that_← normal underscore is hereIt is subtle, but they are different.  Does this really matter?
2023-10-16T20:27:43Z
OK, if anyone else is interested from theSentencePiecedocs that I just discovered:SentencePiece treats the input text just as a sequence of Unicode characters. Whitespace is also handled as a normal symbol. To handle the whitespace as a basic token explicitly, SentencePiece first escapes the whitespace with a meta symbol “▁” (U+2581) as follows.github.comGitHub - google/sentencepiece: Unsupervised text tokenizer for Neural... - Whitespace is treated as a basic symbolmasterUnsupervised text tokenizer for Neural Network-based text generation. - GitHub - google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation.