Not getting substantial training time improvement with LORA - is this expected?
Hi all.I am not getting a substantial training time improvement with LORADetails :My dataset size is ~2M training samplesWhen I am fine-tuning roberta-base model ( base model has ~125M trainable parameters ) training time is roughly 30.5hrsWhen I am fine-tuning roberta-base with LORA ( r=8, lora_alpha=8, ~2M trainable parameters ) training time is roughly 29hrsBatch size, number of GPUs, number of epochs is same for both 1 and 2Are we guaranteed to observe a substantial improvement in training time when using LORA on large datasets ?Can anyone please guide regarding this
2024-10-07T19:32:38Z
I am seeing the same behavior and I wonder if this is expected.