Do I need to dequantization before merging the qlora
In this DPO trainerlinkIt saysAs suggested by [Benjamin Marie](https://medium.com/@bnjmn_marie/dont-merge-your-lora-adapter-into-a-4-bit-llm-65b6da287997), the best option for merging QLoRA adapters is to first dequantize the base model, then merge the adapter. Something similar to [this script](https://github.com/jondurbin/qlora/blob/main/qmerge.py).do we still need to do this merge?  I feel the peft already integrated this feature?
2024-10-04T23:36:41Z
I feel the peft already integrated this feature?I use the latest version of PEFT on pip, not as new as the github version…Maybe PEFT alone support this feature.However, applying LoRA with quantization either doesn’t work, or seems to work, but causes errors during inference.The error is different for diffusers as well as transformers, but almost the same error occurs.I am not sure if this is due to insufficient support in the HF library, or bugs or compatibility issues with PEFT, torch, bitsandbytes, or other libraries, but I too recommend dequantizing and then merging under the current circumstances.
2024-10-05T00:16:10Z
After I finetune the model with lora, and save the model (peftmodel),then I run:base_model = trainer.model.model
base_model_dequantized = qmerge.dequantize_model_2(model=base_model, dtype=torch.bfloat16)
peft_model = PeftModel.from_pretrained(base_model_dequantized, adapter_path)
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained(output_path, safe_serialization=False)it shows this error when do the merging:[rank0]:   File "/opt/miniconda/lib/python3.9/site-packages/peft/tuners/lora/bnb.py", line 344, in merge
[rank0]:     output = dequantize_bnb_weight(weight, state=weight.quant_state)
[rank0]: AttributeError: 'Parameter' object has no attribute 'quant_state'
2024-10-05T00:24:35Z
I this this function in lora merge:peft/src/peft/tuners/lora/bnb.py at a0788a3f92c8220f68d2185aeef0266d6b725bfe · huggingface/peft · GitHub
2024-10-05T00:30:02Z
There are several possible causes, but it seems that there are many bitsandbytes version problems. Not sure if the latest version would work…pip install -U bitsandbytes
2024-10-05T01:43:00Z
I think the reason is that in the functionbase_model_dequantized = qmerge.dequantize_model_2(model=base_model, dtype=torch.bfloat16)it is using this function atlinkSo it is dequantized, but when I call merge_and_unload later, it seems trying to do dequantization again from the link above.This is from this function
2024-10-05T03:57:04Z
How about this?base_model_dequantized = base_model.dequantize()huggingface.cobitsandbytesWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-10-06T07:14:50Z
The reason isbase_model = trainer.model.modelwill not get the base model, the base model is modified in place so it is a lora model now,  so when I runbase_model_dequantized = qmerge.dequantize_model_2(model=base_model, dtype=torch.bfloat16)it is not using base model but lora model.The base model istrainer.model.unload()
2024-10-06T09:24:13Z
Could this be PEFT’s merge_and_unload() calling dequantize_model()?If so, then dequantize would be executed in the following code.base_model = trainer.model.model
#base_model_dequantized = qmerge.dequantize_model_2(model=base_model, dtype=torch.bfloat16)
peft_model = PeftModel.from_pretrained(base_model, adapter_path)
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained(output_path, safe_serialization=False)
2024-10-08T17:21:51Z
The reason is that the trainer.model is still peft model. We can only get base model using unload method.
2024-10-09T04:27:26Z
Hmmm, if that is the case, I wonder what is going on.I think it might work anyway if we can stop the dequantize from being called twice…I think this is a bug…I wonder if the github version of PEFT could fix it?