Where is SageMaker Distributed configured in HF Trainer?
I see that the HF Trainerrun_qa scriptis compatible with SageMaker Distributed Data Parallel, but I don’t see where is it configured?In particular, I can see in thetraining_argsthat smdist gets imported and configured, but where is the model wrapped with smdist DDP?According to thesmdist docthe below snippet is a required step ; I’d like to understand where it’s done with HF Trainerfrom smdistributed.dataparallel.torch.parallel.distributed import DistributedDataParallel as DDP
model = DDP(Net().to(device))
2021-05-06T11:34:57Z
Hey@OlivierCR,both the SageMaker Distributed Data-Parallel and the Model-Parallel library are directly integrated into theTrainerAPI, which uses and initializes both libraries automatically.ForSMD:The library is first imported with an alias for the default PyTorch DDP libraryhereand then wrapps the modelhereP.S. The_wrap_model()function also handlesSMP
2021-05-06T11:50:26Z
wow so amazing, good job guys