Running a Trainer in DistributedDataParallel mode
I am trying to train a model on four GPUs (AWS ml.p3.8xlarge). As far as I can tell, to get my model to train inDistributedDataParallel, I only need to specify some integer value forlocal_rank. But my understanding is that this will only distribute the training across a single GPU (whichever I specify withlocal_rank).What is the proper way to launchDistributedDataParalleltraining across all four GPUs using aTrainer? Do I have to launch something via the command line (as hinted at herehttps://github.com/huggingface/transformers/issues/1651)?
2020-10-24T08:24:39Z
Hi@deppen8Yes, you’ll need to usetorch.distributed.launchfor distributed training.Seethis commandfor an example.