Training with Trainer really slow
HiI am trying out this tutorial here (Iink), which is about training a large language model from scratch, but the training process is really slow with Trainer. For the 6 GB dataset, it is mentioned that it will take about 20 hours for the full dataset, 2 hours for 10% of the dataset, but i see that it’ll take 15 days for training to complete for the whole dataset.I think some part of the code is being executed on CPU, (I see something about iterations per second), and i was wondering if there is any way to speed it up?I am a beginner at this. I have 2x3080TI gpus, and a really slow CPU. 64 gigs of RAM.Thanks