Can accelerator handle the distributed sampler?
As far as I know, for Pytorch, RandomSampler can not be directly used in the distributed data parallel training sinceDistributedSampleris desired (thislinkdiscusses the problem). I am wondering whetheraccelerator.prepare(dataloader)handles the data split for multiple GPUs if I use theRandomSampler, so that the sub-dataset on each device are exclusive.
2021-12-20T13:53:49Z
You don’t have to worry about using a distributed sampler with Accelerate. Whatever your sampler is, Accelerate will automatically shard it for all processes.
2021-12-21T01:47:44Z
That’s great! Thanks!