Using Transformers with DistributedDataParallel — any examples?
Hi!I’ve been consulting this page:https://huggingface.co/transformers/parallelism.html#data-parallelthat says using DDP with transformers is “almost trivial”. But is there an example available?Am I supposed to followhttps://pytorch.org/tutorials/intermediate/ddp_tutorial.htmljust as if I was working with a regular PyTorch model and its optimizer exposed (as opposed to having it abstracted via transformers.Trainer)?Also, I have some Dataset-related questions. I’ve written a custom dataset class that extends torch.Dataset. My dataset class yields samples from stored binary chunks with pre-shuffled pre-tokenized data (to maximize reading speed within a chunk). Therefore, I had to disable Trainer’s shuffling behavior by replacing RandomSampler with SequentialSampler within Trainer._get_train_sampler.Will this hack work with DDP? Would it work if I switched to another distributed backend, like deepspeed? Is there a better way to do this?
2021-10-14T14:46:56Z
You have examples usingAcceleratewhich is our library for distributed training for all tasks in theTransformers repo.As for your hack, you will need to use the distributed version of the SequentialSampler. You might be better off replacing the sampler for the training dataloader by_get_eval_samplerinstead of_get_train_sampler.
2021-10-15T12:45:33Z
Introduction for theAcceleratelibrary says I have to be willing to write a forward loop (forgoing Trainer). Is there a way for me to enable DDP training while continuing using Trainer?Replacing_get_train_samplerwith_get_eval_samplerlooks like a much more elegant solution, thank you!
2021-10-15T12:58:44Z
I misunderstood your question then, but I thought you wanted an example with the model and optimizer exposed. That’s why I pointed you to Accelerate.
2021-10-15T13:01:27Z
Yup! I’d like to keep using Trainer, I’m sorry if I wasn’t clear enough.
2021-10-15T13:08:45Z
Then you just need to properly launch your training script, seehere.
2021-10-15T13:50:07Z
I had to create a--local_ranklaunch argument and pass its value to Trainer, and then it worked. Thanks!
2022-08-17T14:29:33Z
can you share the command you ran? and summarize what you did please?@treeofknowledgethis discussion is slightly incomplete imho. For example, usually we wrap the mdl in DDP to have this type of (distributed) data parallel type of thing to work.Did youwrap the model in DDP?change the args to trainer or trainer args in anyway?wrap the optimizer in any distributed trainer (like cherry? cherry is a pytorch lib for things like this)also, what about the init group that is usually needed?Thanks in advancemade a real question of this here:python - How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? - Stack OverflowHow to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)?
2022-08-17T14:42:12Z
@sguggerthis docEfficient Training on Multiple GPUssuggests that ZeRO might also be used in this case (where only the trainer api is needed/wanted). How would one use ZeRO for in that case and keep the trainer? Do you have any links/demos/notebooks?
2022-08-17T19:12:16Z
In case that wasn’t clear this will do everything automatically:python -m torch.distributed.launch --nproc_per_node 2 ~/src/main_debug.pysee details:How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? - #4 by brando
2022-09-05T03:49:16Z
LOL. I find most questions can be solved by carefully reading the documentation. Thanks for your help~
2023-05-08T06:46:08Z
Hi@sgugger, I’m curios about how Trainer works. After I look at the script, I found that when saving model at checkpoint, the script didn’t uselocal_rankargument to make the script only saving model on first worker. But, the example from Pytorchhereshowing that saving model at checkpoint using parameterlocal_rank. Is it okay to do what the Trainer do?