Inference after QLoRA fine-tuning
I’ve fine-tuned a model via QLoRA by following this notebook:Google ColabAnd I pushed the adapter weights to the hugging face hub.When it comes time to predict with the base model+adapters, should I quantize the base model again (given the adapters were trained alongside a frozen quantized base model)?Or is it valid to load the base model unquantized, attach/merge the adapters as usual, and predict away?## TRAINING
bnb_config = BitsAndBytesConfig(...)
model = AutoModelForCausalLM.from_pretrained(..., quantization_config=bnb_config)  # fit adapters alongside quantized base model
model = prepare_model_for_kbit_training(model)
peft_config = LoraConfig(...)
trainer = SFTTrainer(model=model, peft_config=peft_config, ...)
trainer.train()
trainer.push_to_hub()  # pushes adapter weights only

## INFERENCE
peft_config = PeftConfig.from_pretrained(<hub_id>)
base_model = AutoModelForCausalLM.from_pretrained(peft_config.base_model_name_or_path, device_map="auto")  # should I quantize here as I did when fitting the adapters with QLoRA?
peft_model = PeftModel.from_pretrained(base_model, <hub_id>)
model = peft_model.merge_and_unload()
2023-10-03T20:43:08Z
This should help:kaitchup.substack.comLoRA Adapters: When a Naive Merge Leads to Poor PerformanceThe case of LoRA adapters fine-tuned with QLoRA
2023-11-07T16:31:14Z
Unfortunately lots of the content is behind a paywall… Any other sources?
2023-11-07T17:05:55Z
Apparently, PEFT v.0.6.0 which was released a couple of days ago is now able to to merge LoRA weights with 8bits base models. Should look more into it to make sure it’s done right, but if we have faith it should be fine. Haven’t tried it yet tho
2024-01-12T23:52:30Z
HI@lewisbailsdid you find an answer your question? I have the same question right now.
2024-01-13T10:46:07Z
If you fine-tuned a model using PEFT, then at inference time you can just use the AutoModelForCausalLM class, which will automatically load the base model + adapter weights for you (thanks to thePEFT integrationin Transformers).You can additionally also passload_in_4bit=Trueanddevice_map="auto"in order to do 4-bit inference and automatically place the model on the available GPUs.I’ve made anotebookthat goes over QLoRa fine-tuning of Mistral-7B which also includes an inference section.
2024-04-04T03:53:26Z
you’re a real one brother.
2024-06-05T21:03:18Z
Hello Niels,Thank you for sharing.Did you ever try to compare QLoRA with QA-LoRA in terms of speed and accuracy during the inference time?I would greatly appreciate sharing your experiences.Best,Rohit
2024-06-07T15:11:55Z
hey can you help me?There is an issue while running on my local Mac machine. When changing the model name from “microsoft/DialoGPT-small” to “meta-llama/Meta-Llama-3-8B”, the machine is hanging and no response.GitHub Link