AutoTokenizer is very slow when loading llama tokenizer
Code is like this:tokenizer = LlamaTokenizer.from_pretrained('decapoda-research/llama-13b-hf', use_fast=True)
    tokenizer.save_pretrained('./llama_tok')

    tokenizer = LlamaTokenizer.from_pretrained('./llama_tok/') # very fast to load
    tokenizer = AutoTokenizer.from_pretrained('./llama_tok/') # very slow to loadWhat is the cause of this and how could I fix it please?
2023-06-22T22:34:47Z
I also found this issue.
2023-10-31T05:39:30Z
Same issue here also for loading Vicuna tokenizers. Got it workaround by replacing AutoTokenizer with LlamaTokenizer.