

In the following sections of this guide we dig deeper into how these different parallelism methods work.
Data Parallelism
Even with only 2 GPUs, you can readily leverage the accelerated training capabilities offered by PyTorch's built-in features, 
such as DataParallel (DP) and DistributedDataParallel (DDP). Note that 
PyTorch documentation recommends to prefer 
DistributedDataParallel (DDP) over DataParallel (DP) for multi-GPU training as it works for all models.
Let's take a look at how these two methods work and what makes them different.
DataParallel vs DistributedDataParallel
To understand the key differences in inter-GPU communication overhead between the two methods, let's review the processes per batch:
DDP: