Tokenizer is not being loaded on Huggingface Inference
I was usingthisto finetune a GPT medium.tokenizer = GPT2Tokenizer.from_pretrained('gpt2', bos_token='<|startoftext|>', eos_token='<|endoftext|>', pad_token='<|pad|>') #gpt2-medium
configuration = GPT2Config.from_pretrained('gpt2', output_hidden_states=False)
model = GPT2LMHeadModel.from_pretrained("gpt2", config=configuration)
model.resize_token_embeddings(len(tokenizer))

# after finetuning

model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)But when I upload the model and tokenizer on the model hub (ritwikm/gandhi-gpt), I see the following errorCan't load tokenizer using from_pretrained, please update its configuration: No such file or directory (os error 2)On my local machine, I am loading the same tokenizer and model using the following lines:model = model.from_pretrained(output_dir).to(device)
tokenizer = tokenizer.from_pretrained(output_dir)And it works fine. It just fails on the inference.I have tried many solutions like using only those tokenizer files which are available in the official repo (gpt2).I saw the post by bala1802, even that didn’t help.What can be done?