GPT2Tokenizer.decode maps unicode sequences to the same string '�'
Hello everyone,I have a naive question about tokenizers, particularly GPT2 Tokenizer. I have this encoded a text sentence, and I’ve obtained the token: 29826, which inGPT2Tokenizer Vocabularycorresponds to the Unicode sequence “\u00e6\u0143”.For some reason, I needed to convert 29826 back to its token, i.e., into text, so I used the following code snippet:from transformers import GPT2Tokenizer, AutoModelForCausalLM
import torch
import torch.nn.functional as F

tokenizer = GPT2Tokenizer.from_pretrained( "EleutherAI/gpt-neo-125M")
model = AutoModelForCausalLM.from_pretrained( "EleutherAI/gpt-neo-125M")
tokenizer.pad_token_id = tokenizer.eos_token_idI found myself comparing the logits distribution for token_id 29826 using the code below:# Version 1. No need to encode because we already have access to token_id
token_ids_1 = torch.tensor([[29826]])
logits_1 = model.forward(token_ids_1).logits.squeeze().detach().numpy()

# Version 2. Need to get input_ids
token = tokenizer.decode([29826])
token_ids_2 = tokenizer(token, return_tensors="pt", add_special_tokens=False).input_ids
logits_2 = model.forward(token_ids_2).logits.squeeze().detach().numpy()

## Visualize the logits distribution
import matplotlib.pyplot as plt
import seaborn as sns

sns.histplot(logits_1, label="Logits 1")
sns.histplot(logits_2, label="Logits 2")
plt.legend()Surprisingly, the logit distributions were different (see attached picture). It appears that having thistokenizer.decodeconversion step is actually creating the string  ‘�’, which is being decoded to the index 4210 (a different unicode sequence" \u00ef\u00bf\u00bd").In fact, both expressionstokenizer.decode([4210])andtokenizer.decode([29826])get decoded to the same  ‘�’ character instead of their actual unique expression. But I was hoping that they would decode to their actual unicode string.Is there any way I can deal with this? Is this expected? I’ve tried tweaking the string to decode to its actual unicode string but failed.Environment:transformersversion: 4.26.1tokenizersversion: 0.13.2Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.31Python version: 3.9.16Huggingface_hub version: 0.12.0PyTorch version (GPU?): 1.12.1 (False)Tensorflow version (GPU?): not installed (NA)Flax version (CPU?/GPU?/TPU?): not installed (NA)Jax version: not installedJaxLib version: not installedUsing GPU in script?: FalseUsing distributed or parallel set-up in script?: FalseI executed the code snippets on a Jupyter Notebook (jupyterlab 3.6.1).
2023-02-24T20:46:59Z
I guess the more appropriate way would be to use:tokenizer.convert_ids_to_tokens([4210])
2023-02-24T21:57:46Z
On the use oftokenizer.convert_ids_tokens, I’ve found that. the two expressions below lead to two different strings:ts = tokenizer.encode("Hello!\n I can't do this anymore")
# [15496, 0, 198, 314, 460, 470, 466, 428, 7471]andtokenizer.convert_ids_to_tokens(ts)
# ['Hello', '!', 'Ċ', 'ĠI', 'Ġcan', "'t", 'Ġdo', 'Ġthis', 'Ġanymore']How should I interpret the difference between the two?
2023-03-15T19:36:38Z
Based onhttps://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475:In GPT2 and Roberta tokenizers, the space before a word is part of a word, i.e."Hello how are you puppetter"will be tokenized in["Hello", "Ġhow", "Ġare", "Ġyou", "Ġpuppet", "ter"]. You can notice the spaces included in the words aĠhere. Spaces are converted in a special character (theĠ) in the tokenizer prior to BPE splitting mostly to avoid digesting spaces since the standard BPE algorithm used spaces in its process (this can seem a bit hacky but was in the original GPT2 tokenizer implementation by OpenAI).