How does Accelerate ensure uniqueness of data samples across GPUs?
Hi,I have the following questions about the inner workings of Accelerate. If there is an existing document answering these, please link to it.When using DDP, how does Accelerate sample the data from the dataloader such that one data instance is used exactly once across all the GPUs (i.e. batches used per GPU don’t share examples)? Is there any change in this behaviour when loading from anIterableDataset?When passing the same seed toaccelerate.utils.set_seed()function, is reproducibility guaranteed for DDP training runs? That is, is k-th batch on each GPU same across different training runs?I am using anIterableDatasetfor training in a DDP setting and want to ensure the reproducibility of the training runs. If it does not come out of the box with Accelerate, please guide me on how to achieve the same.Thanks.
2023-06-21T13:59:50Z
mdrpanwar:When using DDP, how does Accelerate sample the data from the dataloader such that one data instance is used exactly once across all the GPUs (i.e. batches used per GPU don’t share examples)? Is there any change in this behaviour when loading from anIterableDataset?Check out our sampler, which splits up the data as we grab it:https://github.com/huggingface/accelerate/blob/main/src/accelerate/data_loader.py(seeBatchSamplerShardandIterableDatasetShard)mdrpanwar:When passing the same seed toaccelerate.utils.set_seed()function, is reproducibility guaranteed for DDP training runs? That is, is k-th batch on each GPU same across different training runs?Yes.
2023-06-21T14:20:12Z
Thanks for your reply and for sharing the reference. To confirm my understanding let me provide an example.Assume two setups:(1) DDP on 2 GPUs, and aper_gpu_batch_sizeof 16(2) Training on a single GPU with a batch size of 32It seems that what the accelerate samplers do for setup (1) is equivalent to setup (2). The sampler just splits the data across GPUs. So, the examples on which the model gets trained are the same in both setups. Is that correct?However, I tried these two setups and the DDP case gives a higher loss than the single GPU setup. What could be the reason for that?