Issue in loading the saved tokenizer
image1481×512 67.2 KB
2024-01-04T13:54:43Z
i was using the following code :from transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained(“Salesforce/xgen-7b-8k-base”,trust_remote_code=True)tokenizer.pad_token = tokenizer.eos_tokentokenizer.save_pretrained(“toksav”)loaded_tokenizer = AutoTokenizer.from_pretrained(“toksav”)