Push 4-bit converted model to hub
Hey there,I am quantizing a model to 4-bit usingBitsandBytes, and when I try to push the model to the hub I am getting the following error:You are calling `save_pretrained` on a 4-bit converted model. This is currently not supportedI’ve pushed a4-bit converted modelto the hub before after finetuning it with peft, however I was wondering whether I can do it without  going through that path.I knowGPTQgives me more options, but it only works for text models.Alternatively I can go through the finetuning and use methods like pruning, distillation  to get the model smaller, but I am wondering if there is work around for pushing a 4-bit converted model to the hub
2023-10-27T07:28:47Z
Following PRs do handle this issue^github.com/TimDettmers/bitsandbytesSave and load  in NF4 / FP4 formatsTimDettmers:main←poedator:save4opened08:29PM - 07 Sep 23 UTCpoedator+353-96Purpose: enable saving and loading transformers models in 4bit formats.
Enables…this PR in transformers: https://github.com/huggingface/transformers/pull/26037

addresses feature request #603 and other similar ones elsewhere.


tested with Bloom-560 and Llama-2-7b. 
tested quantization, saving, loading,  -- matched tensors and quant_states -- matched inference results.
tested both nf4 and fp4
test addedgithub.com/huggingface/transformers[bnb] Let's make serialization of 4bit models possiblehuggingface:main←poedator:save4opened07:37PM - 07 Sep 23 UTCpoedator+194-67## What does this PR do?
Purpose: enable saving and loading transformers models…in 4bit formats.

tested with Bloom-560 and Llama-2-7b. Save - Load - match tensors and quant_states - match inference results.

## connection with bitsandbytes
Requires this PR in bitsandbytes: https://github.com/TimDettmers/bitsandbytes/pull/753 to be able to save/load models

## testing:
the functionality was tested doing this series of commands:
```
model = m4 = transformers.AutoModelForCausalLM.from_pretrained(..., [with 4bit quant config])
model.save_pretrained(SAVE_PATH, safe_serialization= [False / True] )
model2 = transformers.AutoModelForCausalLM.from_pretrained( SAVE_PATH)
# then matching all params and quant_state items between the models, matching inference results
```
Specific tests will be added to this PR once the [bitsandbytes PR](https://github.com/TimDettmers/bitsandbytes/pull/753) merges.

## Open questions to the maintainers:
1. I added and tested code necessary for straightforward save/load operations with LLMs. Yet there may be other kinds of function calls that may need to be updated to handle 4bit save/load - pls suggest if/where to expand this PR.
specifically: this PR covers `load_state_dict_into_meta_model()` but not `load_state_dict_into_model()` for I did not find an  example that uses it. 

2. Some of my edits may not fit with style / refactoring plans of the maintainers - pls give guidance if needed.I will  just wait until they are merged
2024-02-18T10:42:08Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.