`GPT2Tokenizer` Tokenizer handling `\n\n` differently in different settings
Hi all, I was inspecting the tokenization output ofGPT2Tokenizerand observe that it will tokenize\n\nas a single token in some cases whereas in other cases it will tokenize it as two\ntokens. Can someone explain why would the tokenizations differ?Here is a quick example:from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer(" questions.\n\n Human")['input_ids']
>>> [2683, 13, 198, 198, 20490]  #  [' questions', '.', '\n', '\n', 'Human'] if followed by a space, will tokenize as 2 separate '\n'
tokenizer(" questions.\n\nHuman")['input_ids'] 
>>> [2683, 13, 628, 5524] # [' questions', '.', '\n\n', ' Human']
2023-10-03T23:21:41Z
The reason “\n\n” might be tokenized differently in varying contexts could be influenced by a few factors:Whitespace Handling: Tokenizers are often sensitive to whitespace. In your example, when a space follows the “\n\n”, the tokenizer might opt to handle each “\n” separately.Context: The tokenizer may take into account the surrounding characters or tokens when determining how to tokenize a substring. It’s a way to better represent the syntactic and semantic properties of the text.
2023-10-04T00:17:30Z
Thank you for the responseTo follow up I would like to understand of how is the context such as the whitespace come into play in the GPT2 tokenizer when it tokenizes a string?For example it would be great to see the step by step creation of the token list for the simple example I provided
2023-10-04T11:05:44Z
Do you mean like this?Starting Point: Tokenizer sees the string " questions.\n\n Human".

Tokenizing "questions": Recognizes " questions" as a single token [2683].

Tokenizing '.': Next is a period, recognized as [13].

Tokenizing '\n\n' with a following space: The presence of a space after \n\n causes the tokenizer to treat each newline as separate tokens. Thus, we get two tokens for \n as [198, 198].

Tokenizing "Human": The last word " Human" is recognized as a token [20490].If you want to do this programmatically, I suppose that would be a interesting side project.
2023-10-04T16:37:55Z
Hi Thanks again for the response, but this is just stating the obvious…I am interested to understand the rules of the tokenization, so that If I had such rules in hand, it would be clear why when\n\nis followed by a space, it is tokenized differently.  Would be great to see the tokenization rules for the GPT2 tokenizer with a given vocabulary, explained in plain words.