Why is it so slow to access data through iteration with hugginface dataset?
I have a large txt file with one sentence per line.When I try to read this file with the below codes, it takes 0.05sec.with open([data_path],"r") as f:
    test_time=time.time()
    for i in range(100000):
        a=f.readline()

print("%.2f"%(time.time()-test_time))However, when I tried to do with huggingface dataset, It takes 3.2sec.dataset=load_dataset("text",data_files=[data_path])

test_time=time.time()
for i in range(100000):
    a = dataset["train"][i]["text"]
logger.info("time = %.2f"%(time.time()-test_time))Because i want to read large data, it is very slow to access full data with huggingface dataset.I would like to know why this happens.
2022-07-21T12:24:48Z
Hi! This is how the “access through iteration” should look like to correctly compare the performance (your code currently benchmarks the standard indexing, which can’t leverage sequential access present while iterating over rows):for i, ex in enumerate(zip(dataset)):
    if i + 1 == 100000:
        breakAt the moment,Dataset.__iter__callDataset.__getitem__, so the result should still be the same, but I think we can make this faster if we iterate over batches by default.@lhoestqWDYT? From what I remember, this issue has been reported several times, so I think we should address it.
2022-07-21T13:03:03Z
mariosasko:At the moment,Dataset.__iter__callDataset.__getitem__, so the result should still be the same, but I think we can make this faster if we iterate over batches by default.@lhoestqWDYT? From what I remember, this issue has been reported several times, so I think we should address it.Yup I agree it will be much faster to access the data by small batches. We can consider usingpa_table.to_batches()first, and then iterate over the elements one RecordBatch at a time.For further optimizations, we can even re-split each RecordBatch to not be slowed down bybatch.slicethat is linear in time with respect to the size of the RecordBatch.Another idea would be to pass a subbatch of more than 1 element to the formatter, and then unbatch the data, to further reduce the back and forth between python and c++ code, and benefit from batched c++ code for numpy/torch/tf formattingEDIT: oh actually there is apa_table.to_readerthan returns an iterable, it could be simpler this way:for record_batch in pa_table.to_reader(max_chunksize=1):
    pa_subtable = pa.Table.from_batches([record_batch])
    yield format_table(pa_subtable, key=0, formatter=formatter, ...)