GPT loss increasing
Hi! I’m finetuning a GPT-2 model (360M params = size M) on a huge training set (dozens of GB) with a low learning rate (15e-6, or 0.000015). The loss value dips at the start, and then gradually increases for a long time. I haven’t seen it start to fall again yet. Is this behavior normal/expected?graph809×450 24.2 KB
2021-06-21T14:55:44Z
hey@treeofknowledgethis plot suggests your learning rate is too high after 25k training steps so you could try using a learning rate scheduler like the default provided intransformers.Trainer(seehere)