Creating sharded IterableDataset from a list of IterableDatasets?
Given a list of IterableDatasetsdatasets(len = n), each with n_shards == 1, is it possible to create a single IterableDatasetdswithds.n_shards == n? I have a really roundabout way of doing so:def gen(datasets):
            print("Creating generator")
            for idx, ds in enumerate(datasets):
                print(f"Starting dataset {idx}: {ds.name}")
                for item_idx, item in enumerate(ds):
                    yield item
                print(f"Ending dataset {idx}: {ds.name}")
            print("Generator exhausted")
        ds = IterableDataset.from_generator(gen, gen_kwargs={"datasets": datasets})But this code is not ideal as it doesn’t play well with DataLoader when num_workers > 0.
2024-06-30T04:55:37Z
I had the exact same problem. HF’sdatasets.interleave_datasets()can deal with a list ofiterabledatasets but the returnediterabledatasetwill haven_shardsbeing the smallest of the list. In your case it is 1. So the workaround to achieve the goal is to pre-process so that alliterabledatasets in the list have an_shardsofnbefore passing over tointerleave_datasets().
2024-07-02T14:58:57Z
kienman:Given a list of IterableDatasetsdatasets(len = n), each with n_shards == 1, is it possible to create a singleliteblue.usps.govIterableDatasetdswithds.n_shards == n? I have a really roundabout way of doing so:def gen(datasets):
            print("Creating generator")
            for idx, ds in enumerate(datasets):
                print(f"Starting dataset {idx}: {ds.name}")
                for item_idx, item in enumerate(ds): 
                    yield item
                print(f"Ending dataset {idx}: {ds.name}")
            print("Generator exhausted")
        ds = IterableDataset.from_generator(gen, gen_kwargs={"datasets": datasets})But this code is not ideal as it doesn’t play well with DataLoader when num_workers > 0.You can create a sharded IterableDataset by combining datasets using torch.utils.data.chain. This approach works better with DataLoader and multiple workers, avoiding the issues with your generator method.