Hi, I was wondering if is there a way to download only part of the data of a dataset. In my specific case, I need to download only X samples from oscar English split (X~100K samples). When I try to invoke the dataset builder it asks for >1TB of space so I think it will download the full set of da…

Hi, let me just complete this: split='train[10:20]' returns a slice of the data, but it still downloads everything. If your dataset is too big, please use streaming mode. You can also slice your dataset in streaming mode, see the documentation here: Stream

Download only a subset of a split

🤗Datasets

morenolq April 4, 2022, 4:41pm 3

Thank you, can I also use streaming mode to reach the same?

Download a fraction of data from HuggingFace Datasets

Topic		Replies	Views
Loading just part of dataset 🤗Datasets	3	3365	May 5, 2024
Loading a fraction of data 🤗Datasets	5	4369	May 12, 2023
Downloading a portion of parquet files 🤗Datasets	3	468	May 23, 2024
Download only 1 of many parquet file 🤗Datasets	1	85	August 19, 2024
Can I download only the first half of a large dataset? 🤗Datasets	1	290	May 29, 2024

Download only a subset of a split

Related topics