Which data parallel does trainer use? DP or DDP?
I try to search in the doc. But I didn’t find the answer anywhere.Thank you
2022-03-24T12:22:07Z
It depends if you launch your training script withpython(in which case it will use DP) orpython -m torch.distributed.launch(in which case it will use DDP).
2022-08-17T15:03:18Z
perhaps useful to you:Using Transformers with DistributedDataParallel — any examples?