Distributed Training w/ Trainer
Does anyone have an end-to-end example of how to do multi-gpu, multi-node distributed training using the trainer? I can’t seem to find one anywhere.
2021-07-07T17:52:14Z
All the examples using theTrainerrun in multi-gpu multi-node, you just have to use the PyTorch launcher to properly launch a multi-GPU multinode training.
2021-07-07T18:03:40Z
So is there no code adjustments that need to be made, only how the file is launched?
2021-07-07T18:08:41Z
Yes, theTrainerwill deal with all the rest by itself.
2021-07-23T08:33:27Z
Hi I’m trying to run a multi-node training using the Trainer class, for that I run my script with thepython -m torch.distributed.launch --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr="IP" \ --master_port=1234, however, the script doesn’t wait for the master node. Also when I run in the master node the script doesn’t wait for the child node. Should I set up any env variable? The only thing that I’m doing is passing the local_rank to the TrainingArgs.Thanks for the help!
2021-07-26T13:05:20Z
It’s hard to know what the problem could be without seeing the script you are launching.
2022-04-24T09:02:37Z
what do you mean by “use the PyTorch launcher to properly launch a multi-GPU multinode training” ?
2022-05-31T10:10:45Z
It doesnt work with Longformer. Is this expected ?
2022-05-31T13:31:53Z
https://pytorch.org/docs/stable/distributed.html#launch-utility