GPU memory usage of optimizer's states when using LoRA
Hi, recently I am using LoRA(by using peft) + transformers Trainer + DeepSpeed(ZeRO3) to finetune my model (around 7B params). Before this, I tried full-param finetune as well.The wierd thing is that I found LoRA seems not help to reduce the GPU memory usage compared with the full-param finetune method.My question is : when using LoRA, I checkecd that a lot of paramerters’srequires_gradis set toFalse, will this help to reduce the Adam optimizer’s GPU memory usage?  Since I think settingrequires_gradtoFalsemust help reduce the memory usage of gradients tesnor, but the GPU mem usage did not reduce a lot,  so I was wondering if it is because the GPU-mem taken by the optimizer’s states is still as the same as before?  Will the optimizer record the states of model weights which does not need the gradient?
2024-07-05T00:56:41Z
Hi,Sorry for diverging from your question, but I don’t find so much info about this online. How do you merge the adapters resulting from the Lora+Zero3 finetuning back to the base model?
2024-07-05T01:12:10Z
Solved! I finally find out that it is because I didn’t set thegradient_checkpointing=Trueduring my LoRA training, which takes a lot of GPU memory!
2024-07-05T01:13:35Z
Ah I just call the PeftModel’s.merge_and_unloadmethod. It will change the base model’s weights in-place.
2024-07-05T13:14:09Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.