

In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data 
from layer 3 to 4 requires moving it from GPU0 to GPU1, introducing a communication overhead. If the participating 
GPUs are on the same compute node (e.g. same physical machine) this copying is fast, but if the GPUs are distributed 
across different compute nodes (e.g. multiple machines), the communication overhead could be substantially greater.
Following that, layers 4 to 7 work as they would in the original model. Upon completion of the 7th layer, there is often 
a need to send the data back to layer 0 where the labels are (or alternatively send the labels to the last layer). Now the loss can be 
computed and the optimizer can do its work.
Naive Model Parallelism comes several shortcomings:
- All but one GPU are idle at any given moment: if 4 GPUs are used, it's nearly identical to quadrupling the amount of memory of a single GPU, and ignoring the rest of the hardware. 
- Overhead in data transfer between devices:  E.g. 4x 6GB cards will be able to accommodate the same size as 1x 24GB card using naive MP, but a single 24GB card will complete the training faster, because it doesn't have the data copying overhead. But, say, if you have 40GB cards and need to fit a 45GB model you can with 4x 40GB cards (but barely because of the gradient and optimizer states)
- Copying shared embeddings: Shared embeddings may need to get copied back and forth between GPUs.
Now that you are familiar with how the naive approach to model parallelism works and its shortcomings, let's look at Pipeline Parallelism (PP).
PP is almost identical to a naive MP, but it solves the GPU idling problem by chunking the incoming batch into micro-batches 
and artificially creating a pipeline, which allows different GPUs to concurrently participate in the computation process.
The following illustration from the GPipe paper 
shows the naive MP on the top, and PP on the bottom: