Dora training taking 8x time? Why?
Hi, I am enabling the flag of use_dora in LoRAConfig. When I disable it, the training time is 16 hours and when I enable it, the training time shows 122hours. I have kept all other configs same. What is causing this behaviour?Training LLAMA 8b instruct.Following are my lora and trainingArguments:lora_config:target_modules: “q_proj,k_proj,v_proj,o_proj,gate_proj”r: 32lora_alpha: 16lora_dropout: 0.05use_dora: Trueinit_lora_weights: “gaussian”use_rslora: Truefreeze_layers: 0train_params:learning_rate: 0.00003per_device_train_batch_size: 1per_device_eval_batch_size: 4num_train_epochs: 3gradient_accumulation_steps: 8max_grad_norm: 1eval_strategy: “steps”eval_steps: 0.123optim: ‘adamw_8bit’save_steps: 0.123weight_decay: 0.01fp16: truesave_strategy: “steps”warmup_ratio: 0.1logging_steps: 50gradient_checkpointing: falsereport_to: ‘tensorboard’lr_scheduler_type: ‘cosine’save_total_limit: 100ddp_find_unused_parameters: false