Error loading tokenizer from local checkpoint directory
I’m fine-tuning Whisper for a low-resource language (Chichewa) and followingthis tutorial. One change I have made is to provide a local directory to save the model instead of pushing to Hub. When its time to use the fine-tuned model using the pipeline module, I’m getting this error:Can't load tokenizer for '/content/drive/My  Drive/Chichewa-ASR/models/whisper-small-chich/checkpoint-1000. If you were trying to load it from  'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make  sure '/content/drive/My Drive/Chichewa-ASR/models/whisper-small-chich/checkpoint-1000' is the correct path to a directory containing all relevant files for a WhisperTokenizer tokenizer.When I check the model repository used by the tutorial author, I see that it has several files I don’t have in my model checkpoint directory such astokenizer_config.json. I’m just wondering how do I ensure these files are saved in the local model checkpoint directories?
2023-11-01T02:45:20Z
I had the same problem, have you solved it?
2024-04-04T07:12:28Z
@sanchit-gandhican you help with this, I have similar problem
2024-05-13T01:47:49Z
I had same problem,In tried  to copy the tokenizer_config.json file from whisper model output directory to checkpoint directoryThen i  load the model from checkpoint directoryIt worked form me