Can't load tokenizer with added special tokens
Hi! I have an annoying issue that I can’t find answers to anywhere online…Hope someone can help.I am adding a few special tokens to a gpt2 tokenizer using the following code:model_name = “gpt2”model = AutoModelForCausalLM.from_pretrained(model_name)tokenizer = AutoTokenizer.from_pretrained(model_name)special_tokens = {‘eos_token’: “<|endoftext|>”,‘bos_token’: “<|startoftext|>”,‘additional_special_tokens’: ["<|speaker1|>", “<|speaker2|>”]}tokenizer.pad_token = tokenizer.eos_tokentokenizer.add_special_tokens(special_tokens)vocab = tokenizer.get_vocab()model.resize_token_embeddings(len(vocab))I later save the tokenizer using:model_save_name = ‘SARC_gpt2_prefinetune_2.0’tokenizer.save_pretrained(F"/content/drive/MyDrive/Colab Notebooks/saved_models/{model_save_name}")But when I load the model in a different script using:tokenizer_1 = AutoTokenizer.from_pretrained(’/content/drive/MyDrive/Colab Notebooks/saved_models/SARC_gpt2_prefinetune_2.0’)I get this error:AssertionError: Non-consecutive added token ‘<|startoftext|>’ found. Should have index 50260 but has index 50257 in saved vocabulary.Does anyone know what I’m doing wrong here?Thanks in advance