Difference between AutoModelForCausalLM and peft_model.merge_and_unload() for a LoRA model during inference
Hello,I have finetuned a model, with a PEFT configuration and I am trying to load them for inference.I have readin this answerthat loading the model that you have saved:trainer.model.save_pretrained(new_model)using theAutoModelForCausalLM:model = AutoModelForCausalLM.from_pretrained(
    new_model,
    quantization_config=bnb_config,
    device_map=device_map
)would load both the base_model + adapters.However most of the tutorials suggest the use ofmerge_and_unload()when working with peft models:base_model = AutoModelForCausalLM.from_pretrained(
    base_model,
    low_cpu_mem_usage=True,
    return_dict=True,
    device_map=device_map,
    quantization_config=bnb_config,
)

model = PeftModel.from_pretrained(model=base_model, 
                                  model_id=new_model, 
                                  quantization_config=bnb_config,
                                  device_map=device_map,)

model = model.merge_and_unload(safe_merge=True)As far as I understand these 2 blocks of code should yield the same results but when I print the model structures they are nothing alike. Moreover, they yield different results when used for inference - the results of the latter resemble the output of the base model, whereas the results of the former look like the training data that I used.
2024-03-16T15:57:27Z
I have the same question, hope to get some answers too. Many thanks.
2024-08-02T13:57:25Z
Hi@Radmilhave you find a solution for that? I have the same problem with merge_and_unload().