Pushing a quantized (4bit) model on the Hub
Hi. I trained a model in 4bit. After the training, trying to push the model to the Hub, and I get this error message:NotImplementedError: You are callingsave_pretrainedon a 4-bit converted model. This is currently not supported.Although the error message says ‘save_pretrained’ my code is using ‘push_to_hub’:base_model.push_to_hub(/)When is this function be supported? Thank you very much!!
2023-07-11T10:43:36Z
push_to_hub uses save_pretrained in order to save the model in physical format (i suspect) and  then upload to cloud (HG hub)
2023-07-28T05:58:49Z
Thanks for the comment, but that still doesn’t solve the issue. I tried again, same error message. I’m not sure (and doubt) if 4-bit model is still not supported? If so, what am I doing wrong?Can someone please provide answer or insight to this issue?Thanks!
2023-07-28T06:13:51Z
It is supported for sure. My problem was solved after I used peft 0.4.0 I think. Cheers
2023-07-28T14:15:45Z
I have peft==0.4.0 and I get the same error message when trying to save a 4-bit converted model.
2023-07-28T15:58:26Z
The docs aren’t entirely clear, but my read is that 8-bit is possible but 4-bit is not:huggingface.coQuantize 🤗 Transformers modelsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Note that once a model has been loaded in 4-bit it is currently not possible to push the quantized weights on the Hub. Note also that you cannot train 4-bit weights as this is not supported yet. However you can use 4-bit models to train extra parameters, this will be covered in the next section.
2023-10-07T10:14:03Z
Is this issue is closed???I have a same issue with github version package. (peft and transfomers, etc.)NotImplementedError: You are calling save_pretrained on a 4-bit converted model. This is currently not supported.
2024-01-08T12:53:45Z
This issue can probably be closed.Here’s the issue to follow on this:github.com/TimDettmers/bitsandbytesSave and load  in NF4 / FP4 formatsTimDettmers:main←poedator:save4opened08:29PM - 07 Sep 23 UTCpoedator+376-95Purpose: enable saving and loading transformers models in 4bit formats.
Enables…this PR in transformers: https://github.com/huggingface/transformers/pull/26037

addresses feature request #603 and other similar ones elsewhere.


tested with Bloom-560 and Llama-2-7b. 
tested quantization, saving, loading,  -- matched tensors and quant_states -- matched inference results.
tested both nf4 and fp4
test added
2024-01-08T14:49:14Z
This is now possible, see Github gist:push bnb 4 bit models on the hub · GitHubTweet:https://twitter.com/younesbelkada/status/1739244971905966380
2024-01-08T18:28:20Z
Yeah, correct, with the caveat that you still need to install transformers from source as the latest stable release doesn’t support it. Hence why I tagged the issue above, cheers.