Convert_tokens_to_ids produces <unk>
Hi. I am trying to tokenize single words with a Roberta BPE Sub-word tokenizer. I was expecting to have some words with multiple ids, but when that is supposed to be the case, the methodconvert_tokens_to_idsjust returns<unk>. However,__call__from tokenizer produces the multiple ids. To reproduce the problem, run:from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
token_id = tokenizer.convert_tokens_to_ids("exam")
print(f"{token_id} => {tokenizer.decode([token_id])}")
token_ids = tokenizer("exam").input_ids[1:3]
print(f"{token_ids} => {tokenizer.decode(token_ids)}")Is there a way to makeconvert_tokens_to_idshave the same behaviour astokenizer(token).input_ids[1:3]? THanks in advance for any help you can provide.
2022-10-25T20:01:46Z
I think you misunderstandtokenizer.convert_tokens_to_ids(). Please note this function is to maptokentoid, howeverexamisnotatoken, it is awordinstead. You can check by the following code:tokenizer.convert_ids_to_token([3463, 424])
> ['ex', 'am'] # exam is tokenized to two token!So, it’s obivously that there is no tokenexamin vocab, but[UNK]instead.