Inference API for fine-tuned model not working: No package metadata was found for bitsandbytes
I have fine-tuned a model using QLoRA (4-bit) and then pushed it to the hub. Then, I tried to test it with inference API, but I get the following error:No package metadata was found for bitsandbytesThe model:base model is meta-llama/Llama-2-7b-chat-hffine-tuned on a custom dataset of 50 samples (I am just testing around)I made the model public and can be foundherethisis the colab notebook I used to train it. Note that, after the QLoRA training, I merged the adapter model with the base model, THEN pushed the result to the hub. So the model in the hub is a transformer model. Specifically, atransformers.models.llama.modeling_llama.LlamaForCausalLMthisis a colab notebook that can be used for testing. Note that the test works for the base modelmeta-llama/Llama-2-7b-chat-hfMy suspect is that the docker container behind the inference API does not know that it needs to install bitsandbytes. Is there a way to “tell it”? maybe a tag in the README?
2024-03-02T16:02:28Z
Getting the same error, have you found anything related to the issue?
2024-03-02T16:16:29Z
Unfortunately not. Currently I just gave upBut let me know if you find something.
2024-03-02T17:28:36Z
yeah, same. I tried including requirements.txt, every pip install in that text file, but then too the same error. Don’t know how to figure it out.
2024-03-09T14:50:23Z
I have the same problem. Might this be an issue on the server side? I hope someone from HF looks into it.
2024-03-10T11:04:58Z
facing the same issue here.
2024-03-17T21:52:15Z
Did anyone find a solution for this?
2024-03-18T02:27:10Z
Same Issue
2024-03-18T13:52:46Z
The same here. Any advice?
2024-03-18T19:53:04Z
Hello, I had a similar issue recently.First, make sure to have a requirements file with the packages listed.Also, check that you have a torch version compatible with bitsandbytes.huggingface.coInstallationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Finally, if working with docker, you can either specify it into the requirements file or directly in the docker file by writing something like:RUN pip install -U bitsandbytesYou can check locally by building the image with docker run to check how the building goes.
2024-04-30T02:27:25Z
I have the same problem and haven’t found any solutions. Did anyone find any solutions? This is so frustrating
2024-05-02T16:19:50Z
I went through the exact same process (fine-tune, merge, push).Same problem here and same error using inference API.I have also tried to deploy an inference endpoint and getting another error.Callingcuda()is not supported for4-bitor8-bitquantized models.Not sure what is going on here.
2024-05-13T16:02:40Z
Same issue. Have a requirements.txt file installed as well.
2024-05-16T16:20:39Z
Same issue , same message on inference API (No package metadata was found for bitsandbytes). Tried to add transformers in requirements.txt, not working neitherimport torchimport transformersmodel_id = “meta-llama/Meta-Llama-3-8B”nf4_config = transformers.BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type=“nf4”,bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=torch.bfloat16)model_nf4 = transformers.AutoModelForCausalLM.from_pretrained(model_id,quantization_config=nf4_config)model_nf4.push_to_hub(“intelpen/Meta-llama-3-8B-BB4-v3”)
2024-06-24T19:47:00Z
same issue here as well