DistilBert tokenization does not work as expected
Hi.I’m enriching DistilBert tokenizer with new tokens from new corpus. DistilBert uses WordPiece tokenizer, and based on theHuggingface NLP course, the inference is done by finding the “longest possible token” from the beginning of the word, splitting it, and doing the same for the rest of the word.In my tokenizer, however, I haveinspect,insp,ec,##ec,##ttokens, but when tokenizinginspect, the tokenizer comes up with the following tokens:['insp', 'ec', '##t'].I would expect the tokenizer to return only one token:'inspect'. Even if it splits, I would expect it to return at least['insp', '##ec', '##t'].Is this a bug or some of the part of my code is incorrect?Minimum working example:>> from transformers import AutoTokenizer

>> model_checkpoint = 'elastic/distilbert-base-uncased-finetuned-conll03-english'
>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

>> ('inspect' in tokenizer.vocab, 'insp' in tokenizer.vocab, 'ec' in tokenizer.vocab, '##ec' in tokenizer.vocab, '##t' in tokenizer.vocab)
# (True, False, True, True, True)
>> tokenizer.convert_ids_to_tokens(tokenizer.encode('inspect'))
# ['[CLS]', 'inspect', '[SEP]']

>> tokenizer.add_tokens(['insp'])
# 1
>> ('inspect' in tokenizer.vocab, 'insp' in tokenizer.vocab, 'ec' in tokenizer.vocab, '##ec' in tokenizer.vocab, '##t' in tokenizer.vocab)
# (True, True, True, True, True)
>> tokenizer.convert_ids_to_tokens(tokenizer.encode('inspect'))
# ['[CLS]', 'insp', 'ec', '##t', '[SEP]']