Diffusers text-to-image finetuning example fails on multi-node
I’m trying to finetune stable diffusion 1.4/1.5 for a custom dataset base on diffusers’s example -diffusers/train_text_to_image.py at main · huggingface/diffusers · GitHubFollowing the original script, I used accelerate to use multiple GPUs.losscurve1410×888 95.5 KBWhen I train it locally (2 gpus - pink curve), it converges after around 15K epoch. When I trained it on multiple nodes (4 nodes x 8 gpus - gray curve), it didn’t converge. The hyperparameters except for batch/number of steps remain the same. Do you have any idea why the fine-tuning script fails on multi-node? I suspect that EMA might need some explicit sync between nodes, but I’m not an expert on this.Thanks in advance for any help/advice!
2023-01-03T10:16:57Z
Hi@j-min! Would you mind opening an issue inthe diffusers reposo the team can look into this?Thanks a lot!
2023-03-30T16:36:51Z
I’ve been running on an 8x A100 Google machine and would like to bump it up to 16 using the same script (ideally). Could you point to the issue you opened or share any script changes and/or Accelerate config settings you used? Any help or guidance would be much appreciated.