Run_summarization.py + huge dataset
How can I use iterator to  run ((transformers\examples\pytorch\summarization\run_summarization.py)) with huge data files (in JSON Lines format, size> 20 GB) to avoid loading the whole file into the memory? Any suggestions how to use the code with large size JSON files?Framework:Transformers 4.20.1Pytorch 1.11.0+cu113Datasets 2.3.2Tokenizers 0.12.1
2023-03-01T16:08:15Z
I would split the json file in 2 or 3 parts and do the training in 2 or 3 batches. You also have the batch size --device_train_batch_size maybe you can play with it.