DataLoader from accelerator samples from beginning of dataset for last batch
After quite a bit of digging, I realized that when I calldataloader, model = accelerator.prepare(
        dataloader, model
)Theeven_batcheshyperparameter defaults to True, and this is expected behavior in that case. That is, my last batch cannot be split evenly over the number of devices, so a few samples are taken from the start of the dataset. However:When I change my accelerator withaccelerator = Accelerator(even_batches=False)this does not do anything! I can confirm thateven_batchesis set to False, and I get no warning about this.Digging a bit more, thedataloaderreturned by prepare is actually aDataLoaderDispatcher, which does not seem to care about theeven_batchesargument. I can seeherethat it simply rounds up the batch size, andherethat it then samples from the first batch.So, how can I get the accelerated dataset to stop doing this? Can the docs be updated to reflect the fact that settingeven_batchesto False does nothing (I suspect with certain buried dataset types, etc. it does something, not sure though)?
2024-01-15T17:52:58Z
It may be useful for you.ddp_model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) 

with accelerator.join_uneven_inputs([ddp_model], even_batches=False):

        for input, output in dataloader:
                outputs = model(input)
                loss = loss_func(outputs)
                loss.backward()
                optimizer.step()
                optimizer.zero_grad()