Regarding On-the-fly Data Loading
Possible duplicate

#1776 https://github.com/huggingface/datasets/issues/

really looking PR for this feature Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing). Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ? Hi @lhoestq,

I apologize for the late response. This answers my question. Thanks a lot.
Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan