While reading the literature on this topic you may encounter the following synonyms: Sharded, Partitioned.
If you pay close attention the way ZeRO partitions the model's weights - it looks very similar to tensor parallelism 
which will be discussed later. This is because it partitions/shards each layer's weights, unlike vertical model parallelism 
which is discussed next.
Implementations:

DeepSpeed ZeRO-DP stages 1+2+3
Accelerate integration 
transformers integration