

DP:
For each batch:
   1. GPU 0 reads the batch of data and then sends a mini-batch to each GPU.
   2. The up-to-date model is replicated from GPU 0 to each GPU. 
   3. forward is executed, and output from each GPU is sent to GPU 0 to compute the loss.
   4. The loss is distributed from GPU 0 to all GPUs, and backward is run. 
   5. Gradients from each GPU are sent to GPU 0 and averaged. 
Key differences include:
1. DDP performs only a single communication per batch - sending gradients, while DP performs five different data exchanges per batch.
DDP copies data using torch.distributed, while DP copies data within 
the process via Python threads (which introduces limitations associated with GIL). As a result, DistributedDataParallel (DDP) is generally faster than DataParallel (DP) unless you have slow GPU card inter-connectivity.
2. Under DP, GPU 0 performs significantly more work than other GPUs, resulting in GPU under-utilization. 
3. DDP supports distributed training across multiple machines, whereas DP does not.
This is not an exhaustive list of differences between DP and DDP, however, other nuances are out of scope of this guide.
You can get a deeper understanding of these methods by reading this article.
Let's illustrate the differences between DP and DDP with an experiment. We'll benchmark the differences between DP and 
DDP with an added context of NVLink presence:  