

While it may appear complex, it is a very similar concept to DataParallel (DP). The difference is that instead of 
replicating the full model parameters, gradients and optimizer states, each GPU stores only a slice of it. Then, at 
run-time when the full layer parameters are needed just for the given layer, all GPUs synchronize to give each other 
parts that they miss.
To illustrate this idea, consider a simple model with 3 layers (La, Lb, and Lc), where each layer has 3 parameters. 
Layer La, for example, has weights a0, a1 and a2:
La | Lb | Lc
---|----|---
a0 | b0 | c0
a1 | b1 | c1
a2 | b2 | c2
If we have 3 GPUs, ZeRO-DP splits the model onto 3 GPUs like so: