Using this principle, we can update a multi-layer perceptron of arbitrary depth, without the need for any synchronization 
between GPUs until the very end, where we need to reconstruct the output vector from shards. The Megatron-LM paper authors 
provide a helpful illustration for that:

Parallelizing the multi-headed attention layers is even simpler, since they are already inherently parallel, due to having 
multiple independent heads!