How to load a model fine-tuned with QLoRA
I fine-tuned a mistral 7b with QLoRA that initiated as the following code during training:## TRAINING
bnb_config = BitsAndBytesConfig(  
    load_in_4bit= True,
    bnb_4bit_quant_type= "nf4",
    bnb_4bit_compute_dtype= torch.bfloat16,
    bnb_4bit_use_double_quant= True,
)

model = AutoModelForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
)

model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing = False)
peft_config = LoraConfig(...)
model = get_peft_model(model, peft_config)In inferences, I tried to load the model using the same setting as training:## INFERENCING
bnb_config = BitsAndBytesConfig(  
    load_in_4bit= True,
    bnb_4bit_quant_type= "nf4",
    bnb_4bit_compute_dtype= torch.bfloat16,
    bnb_4bit_use_double_quant= True,
)

base_model = AutoModelForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(base_model, "saved-adapter-path").eval()However, the inference process is very slow.I found that loading the model only with load_in_4bit=True like the following. The inference process is much faster.base_model = AutoModelForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    bnb_config = BitsAndBytesConfig(load_in_4bit= True)
)
model = PeftModel.from_pretrained(base_model, "saved-adapter-path").eval()Isn’t using double quant and bfloat supposed to be faster and use less memory when compared to the default setting?What should be the correct way to load the model in my case?
2024-03-19T20:17:04Z
Hi,Refer to mydemo notebookon fine-tuning Mistral-7B, it includes an inference section.In summary, one can simply use the Auto classes (like AutoModelForCausalLM) to load models fine-tuned with Q-LoRa, thanks to thePEFT integration in Transformers. It will automatically load the base model + adapter weights.Optionally, you can then call themerge_and_unload()method on it to merge the adapters into the base model.
2024-07-29T21:01:11Z
Hi please how do i perform the uploading of the model to the hub where it can be downloaded and used by others