Multi-node training
Hello,Thank you very much for theacceleratelib.We are currently experiencing a difficulty and were wondering if this could be a known case.We want to run a training withaccelerateanddeepspeedon 4 nodes with 4 GPUs each. However, we see in our logs that 4 processes consider to be both amain_processand alocal_main_process. We would have expected to see 1main_processand 4local_main_process.Is what we expected wrong? Do you see a mistake we could have made?Thanks a lot in advance!
2022-09-06T07:08:29Z
For someone who may encounter this problem in the future. This was a mistake on our part! We had not specified thedeepspeed_multinode_launcherargument in the accelerate configuration, so it was set toNone. Addingdeepspeed_multinode_launcher: standardinside thedeepspeed_configsection to the configuration file solved our problem.
2023-01-16T19:15:50Z
Hi@SaulLu, I am trying to do the same thing but with two nodes, each with 3 GPUs. Could you please share your config file content here?