Trainer API for Model Parallelism on Multiple GPUs
According to themain pageof the Trainer API,“The API supports distributed training on multiple GPUs/TPUs, mixed precision throughNVIDIA Apexand Native AMP for PyTorch.”It seems like a user does not have to configure anything when using the Trainer class for doing distributed training. However, I am not able to find which distribution strategy this Trainer API supports. From my understanding, the Trainer class automatically uses DDP when multiple GPUs are detected. And if a user wants to also distribute model parameters onto different GPUs, then we have to pass in configurations for FSDP or deepspeed.Can someone confirm that my understanding is correct or not please?
2023-08-04T14:22:31Z
zcakzhu:Can someone confirm that my understanding is correct or not please?It depends on how you launch the script. If you usetorch.distributed.launch(or haveaccelerate configsetup for multi-gpu) it’ll use DistributedDataParallism. To use model parallelism just launch withpython {myscript.py}and it should pick up model parallism. (If you find it does not, or need some more assistance, let me know!)You can verify if so by checking iftrainer.args.parallel_modeprintsParallelMode.NOT_DISTRIBUTED.
2023-08-04T14:38:23Z
Thanks for your reply! It is super helpfulIt is great to know that by just runningpython {myscript.py}the class will use model parallelism.A follow-up question from me is, how is the Trainer’s model parallelism differ from Deepspeed and FSDP? Is there any documentation that I can read into to gain more knowledge of what is happening at the backend?Thanks a lot!
2023-08-04T16:09:04Z
zcakzhu:how is the Trainer’s model parallelism differ from Deepspeed and FSDP? Is there any documentation that I can read into to gain more knowledge of what is happening at the backend?It just uses raw pytorch model parallism at that point, which is the equivalent of doing.to(device_num). here’s a good doc discussing it:Model Parallelism — transformers 4.7.0 documentation
2023-08-05T12:53:50Z
is it enough to install deepspeed (pip3 install deepspeed) and runaccelerate configand sayyeswhen asked aboutdeepspeed, to getdeepspeedwithTrainer()?
2024-09-10T12:12:55Z
Hello,@muellerzrI would like to see the inner process of how huggingface Trainer activatesmodel parallelismby just givingis_model_parallelargument to True.I checked that this argument makesplace_model_on_device=False intransformers/src/transformers/trainer.py at main · huggingface/transformers · GitHub.However, I don’t get what happens in the next step. Are there some codes representing the division of model parameters ? It would be great if I could get your help.Thank you.