LLaMATokenizer renders this incompatible with huggingfaces transformers pipeline
#1
by
Alignment-Lab-AI
- opened
in transformers the tokenizer is listed as LlamaTokenizer
attempting to run this through any transformers based system results in an error, rendering it incompatible unless the user downloads it and renames it themselves.
i believe attempting to even download the model from huggingface throws the error, its been a bit over a week since i involved BigTrans in my research but i believe i had to wget each file link to retrieve it