Attribute

We have 10 batches of 512 length. If we parallelize them by attribute dimension into 2 devices, 10 x 512 will be 10 x 2 x 256.

Parameter

It is similar with tensor model parallelism or naive layer-wise model parallelism.