Transformer model parallel does not work with Pytorch DDP for multi-node training
Thenaive model parallelismin Transformer models such as T5 does not work with pytorch Distributed Data Parallel for multi-node training. It will freeze at the step of torch.nn.parallel.DistributedDataParallel(Model). I followthis instructionto write the code. Anyone met similar situations?