Accelerate Multi-GPU on several Nodes How to
Hi,I wonder how to setup Accelerate or possibly train a model if I have 2 physical machines sitting in the same network.  Each machine has 4 GPUs.Can I use Accelerate + DeepSpeed to train a model with this configuration ?Can’t seem to be able to find any writeups or example how to perform the “accelerate config”.Thanks.
2021-10-13T11:38:55Z
I’m not sure what documentation you need, just typeaccelerate configin the terminal of both machines and follow the prompts.
2021-10-13T12:02:29Z
Hi,I have read the doc fromaccelerate.  This doc shows how I can perform training on a single multi-gpu machine (one machine) using the “accelerate config”.I am looking for example, how to perform training on 2 multi-gpu machines.  In other words, in my setup, I have 4 x GPU per machine.What are the packages I needs to install ?  For example:machine 1, I install accelerate & deepspeed.  Run  accelerate configmachine 2, do I also just install accelerate & deepspeed ?Is the training on multi-gpu using 2 machines possible ?
2021-10-13T12:55:39Z
Like I said, you need to runaccelerate configon both machines (and yes you need to install everything you need on both of them), then runaccelerate launch training_script.pyon both machines as well.