Vocabulary count mismatch when loading the previously created tokenizer
I am currently fine-tuning the Wav2Vec2 XLS-R model. I have created a tokenizer that is stored in one of the huggingface repos. Here is how I store my tokenizer in huggingface:from transformers import Wav2Vec2CTCTokenizer

tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("./", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
tokenizer.push_to_hub(repo_name)At a certain time, I want to do the same task using the tokenizer. I load the tokenizer with the following code:tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(repo_name)with that code, all the files needed for the tokenizer are loaded, such asvocab.json,tokenizer_config.json, andspecial_tokens_map.json. All the vocabularies I should have are 30, according to what’s invocab.json.But, here’s something strange. when I load the tokenizer, I get two additional tokens, which are already in my vocabulary. From the image is the tokenizer object information that I loaded on the repo.how did this happen? did I load the tokenizer incorrectly?image1295×185 20.7 KB