SFTTrainer takes up so much ram that it breaks an A100 GPU
Hello folks,I have been trying to fine-tune Llama 3 with VeRA adapter on a quite small dataset, which is"mlabonne/guanaco-llama2-1k". I put my training configs in aSFTConfigand initiated aSFTTrainerobject as my trainer. This is where the code blows up! It couldn’t even pass totrainer.train()stage since the trainer object (initiated withSFTTrainer) took so much RAM (not GPU RAM) that it caused both A100 and T4 to throw OOM.You can find myGoogle Colab notebookhere. I have triedlots of different combination with data types,datasets with different sizes,different quantization and attention implementation configs,using TrainingArguments instead of SFTConfig,different VeRA configsbut nothing worked. I would be more than glad if you take a look at it and give me a glimmer of hope. Thanks!