GPT model loss diverges while fine-tuning
I’m trying to fine-tune the distilled GPT model on a new dataset and I’m having issues with the loss diverging during training. Can anyone thing of why this might be happening? I’ve lowered the learning rate to1e-7which feels extremely low, and the issue persistsScreen Shot 2022-02-11 at 2.24.28 AM502×1420 194 KB