how does this load using ezrknpu?
#1
by
developerbayman
- opened
i know how to load the models from here : https://huggingface.co/Pelochus/ezrkllm-collection
but im not sure about this as it uses transformers ...the thing is i never was able to get transformers to work with the rk3588 NPU ....couldnt get anything to work on the NPU aside from ezrknpu ...this looks the same with .rkllm but all the files confuses me as i just know how to use "rkllm<path to .rkllm model> ...then it works
I just use Transformers for the tokenizer. Here is a Gradio interface: https://github.com/c0zaut/RKLLM-Gradio
You only need to use 1 rkllm file - they are the same model, just with different quant settings.
Video tutorial: https://youtu.be/sTHNZZP0S3E?si=pYze1xtkpWpARssH