Using deepspeed script launcher vs accelerate script launcher for TRL
I’ve been trying to figure out the nature of the deepspeed integration, especially with respect to huggingface accelerate.It seems that the trainer uses accelerate to facilitate deepspeed. But when I look at the documentation, it seems that we still use deepspeed as the launcher, or the pytorch distributedeepspeed --num_gpus=2 your_program.py <normal cl args> --deepspeed ds_config.jsonorpython -m torch.distributed.launch --nproc_per_node=2 your_program.py <normal cl args>But it didn’t mention using the accelerate launcher. I’m confused by this since trainer uses accelerate to facilitate the deepspeed integration.As for the TRL library, it seems that it’s using the accelerate library for it’s trainers as well, but for the TRL library, the official way to launch it is to use the accelerate launcheraccelerate launch --config_file=examples/accelerate_configs/deepspeed_zero{1,2,3}.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_scripthuggingface.coTraining customizationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coExamplesWe’re on a journey to advance and democratize artificial intelligence through open source and open science.I’m wondering if it’s still exchangeable with the deepspeed launcher, of it not, what’s the nature of the facilitation.
2024-01-24T15:53:39Z
It seems that the trainer uses accelerate to facilitate deepspeed. But when I look at the documentation, it seems that we still use deepspeed as the launcher, or the pytorch distributedeepspeed --num_gpus=2 your_program.py <normal cl args> --deepspeed ds_config.jsonorpython -m torch.distributed.launch --nproc_per_node=2 your_program.py <normal cl args>You can still launch the script with the deepspeed and pytorch distribute command. However, since the trainer refacto, the trainer backend reliestotallyon accelerate. The accelerate launch command is there to simplify your life since you only need to remember one command + easy to set up the parameters usingaccelerate config.But it didn’t mention using the accelerate launcher. I’m confused by this since trainer uses accelerate to facilitate the deepspeed integration.Check the dochere! But I agree with you, this section is well hidden.As for the TRL library, it seems that it’s using the accelerate library for it’s trainers as well, but for the TRL library, the official way to launch it is to use the accelerate launcherTRL Trainer relies on transformers Trainer. This is why you can use accelerate there as well. We should probably update more our doc to emphasize on accelerate indeed.I’m wondering if it’s still exchangeable with the deepspeed launcher, of it not, what’s the nature of the facilitation.Under the hood, accelerate launch uses the deepspeed launcher. Same for pytorch distributed.
2024-01-24T19:57:51Z
Thanks for the detailed explanationUnder the hood, accelerate launch uses the deepspeed launcher. Same for pytorch distributed.Ah, that clears up a lot for me. For ‘Same for pytorch distributed.’, does that mean accelerate launch uses pytorch distributed? Or pytorch distributed uses the deepspeed launcher?
2024-01-24T20:34:28Z
SantoshScienceIO:does that mean accelerate launch uses pytorch distributed?Yes, you can see it in this linehere. Also see pytorchdoc.
2024-01-25T08:56:05Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.