

Special considerations: TP requires very fast network, and therefore it's not advisable to do TP across more than one node. 
Practically, if a node has 4 GPUs, the highest TP degree is therefore 4. If you need a TP degree of 8, you need to use
nodes that have at least 8 GPUs.
This section is based on the original much more detailed TP overview.
by @anton-l.
Alternative names:
- DeepSpeed calls it tensor slicing
Implementations:
- Megatron-LM has an internal implementation, as it's very model-specific
- parallelformers (only inference at the moment)
- SageMaker - this is a proprietary solution that can only be used on AWS.
- OSLO has the tensor parallelism implementation based on the Transformers.
SageMaker combines TP with DP for a more efficient processing.
🤗 Transformers status:
- core: not yet implemented in the core
- but if you want inference parallelformers provides this support for most of our models. So until this is implemented in the core you can use theirs. And hopefully training mode will be supported too.
- Deepspeed-Inference also supports our BERT, GPT-2, and GPT-Neo models in their super-fast CUDA-kernel-based inference mode, see more here
🤗 Accelerate integrates with TP from Megatron-LM.
Data Parallelism + Pipeline Parallelism
The following diagram from the DeepSpeed pipeline tutorial demonstrates 
how one can combine DP with PP.