Parallelization strategy for a multi-Node / multi-GPU setup

When you have fast inter-node connectivity (e.g., NVLINK or NVSwitch) consider using one of these options:

ZeRO - as it requires close to no modifications to the model
A combination of PipelineParallel(PP) with TensorParallel(TP) and DataParallel(DP) - this approach will result in fewer communications, but requires significant changes to the model

When you have slow inter-node connectivity and still low on GPU memory: