

At the bottom of the diagram, you can observe that the Pipeline Parallelism (PP) approach minimizes the number of idle 
GPU zones, referred to as 'bubbles'. Both parts of the diagram show a parallelism level of degree 4, meaning that 4 GPUs 
are involved in the pipeline. You can see that there's a forward path of 4 pipe stages (F0, F1, F2 and F3) followed by 
a backward path in reverse order (B3, B2, B1, and B0).
PP introduces a new hyperparameter to tune - chunks, which determines how many data chunks are sent in a sequence 
through the same pipe stage. For example, in the bottom diagram you can see chunks=4. GPU0 performs the same 
forward path on chunk 0, 1, 2 and 3 (F0,0, F0,1, F0,2, F0,3) and then it waits for other GPUs to do complete their work. 
Only when the other GPUs begin to complete their work, GPU0 starts to work again doing the backward path for chunks 
3, 2, 1 and 0 (B0,3, B0,2, B0,1, B0,0).
Note that this is the same concept as gradient accumulation steps. PyTorch uses chunks, while DeepSpeed refers 
to the same hyperparameter as gradient accumulation steps.
Because of the chunks, PP introduces the notion of micro-batches (MBS). DP splits the global data batch size into 
mini-batches, so if you have a DP degree of 4, a global batch size of 1024 gets split up into 4 mini-batches of 
256 each (1024/4). And if the number of chunks (or GAS) is 32 we end up with a micro-batch size of 8 (256/32). Each 
Pipeline stage works with a single micro-batch at a time. To calculate the global batch size of the DP + PP setup, 
use the formula: mbs * chunks * dp_degree (8 * 32 * 4 = 1024).
With chunks=1 you end up with the naive MP, which is inefficient. With a large chunks value you end up with 
tiny micro-batch sizes which is also inefficient. For this reason, we encourage to experiment with the chunks value to 
find the one that leads to the most efficient GPUs utilization.
You may notice a bubble of "dead" time on the diagram that can't be parallelized because the last forward stage 
has to wait for backward to complete the pipeline. The purpose of finding the best value for chunks is to enable a high 
concurrent GPU utilization across all participating GPUs which translates to minimizing the size of the bubble.
Pipeline API solutions have been implemented in:
- PyTorch
- DeepSpeed
- Megatron-LM
These come with some shortcomings:
- They have to modify the model quite heavily, because Pipeline requires one to rewrite the normal flow of modules into a nn.Sequential sequence of the same, which may require changes to the design of the model.
- Currently the Pipeline API is very restricted. If you had a bunch of Python variables being passed in the very first stage of the Pipeline, you will have to find a way around it. Currently, the pipeline interface requires either a single Tensor or a tuple of Tensors as the only input and output. These tensors must have a batch size as the very first dimension, since pipeline is going to chunk the mini batch into micro-batches. Possible improvements are being discussed here https://github.com/pytorch/pytorch/pull/50693
- Conditional control flow at the level of pipe stages is not possible - e.g., Encoder-Decoder models like T5 require special workarounds to handle a conditional encoder stage.
- They have to arrange each layer so that the output of one layer becomes an input to the other layer.
More recent solutions include:
- Varuna
- Sagemaker
We have not experimented with Varuna and SageMaker but their papers report that they have overcome the list of problems 
mentioned above and that they require smaller changes to the user's model.
Implementations:
- PyTorch (initial support in pytorch-1.8, and progressively getting improved in 1.9 and more so in 1.10). Some examples
- DeepSpeed
- Megatron-LM has an internal implementation - no API.
- Varuna
- SageMaker - this is a proprietary solution that can only be used on AWS.
- OSLO - this is implemented based on the Hugging Face Transformers.
🤗 Transformers status: as of this writing none of the models supports full-PP. GPT2 and T5 models have naive MP support. 
The main obstacle is being unable to convert the models to nn.Sequential and have all the inputs to be Tensors. This 
is because currently the models include many features that make the conversion very complicated, and will need to be removed to accomplish that.
DeepSpeed and Megatron-LM integrations are available in 🤗 Accelerate
Other approaches:
DeepSpeed, Varuna and SageMaker use the concept of an Interleaved Pipeline