Loss becoming nearly zero in first 5K steps when training LM from scratch
I am training the ALBERT LM model from scratch.I have already trained it for Hindi and Bangla and it was working fine but when I am training on Gujarati and Telugu, the loss is becoming zero in 5K steps.What could be the reason for the sudden drop in the loss? Can anyone suggest what could be the cause or how to debug such issue?Any suggestions?
2020-07-30T13:45:03Z
Usually, this means you are training on your validation data, so I’d triple-check your training set and validation set don’t contain the same texts, or that there is no leak from one to the other.
2020-07-30T13:53:15Z
I am not using any validation data. I have masked the raw corpus  after sentencpiece tokenization and put it to training.
2020-07-30T13:54:45Z
Then it’s just the model learning your training set. If you really want to know how it would fare on new data, you need to use a validation set and compute the loss on it.
2020-07-30T14:04:09Z
Have used eval set for hindi, And it was taking lot of time to for evaluation so I went without eval data.
2020-07-31T10:59:11Z
But without eval data you don’t know how your model actually perform. You may be overfitting the training dataset
2020-07-31T18:02:19Z
But without eval data you don’t know how your model actually performs. You may be overfittumg
2020-08-01T06:36:55Z
agreed. But what can be cause of sudden drop in loss?
2020-10-16T05:13:29Z
hi, have you resolved your issue. I have a similar problem
2022-04-01T08:17:59Z
@008karan@dineshI have a similar problem (here the post). Have you resolved your issue?@sguggerI’m using validation set too, but train loss becames zero in 4 epochs.I tried to continue the training with others 4 epochs with lower learning rate, but the train loss starts at zero, so I have no improvement.
2023-03-18T10:27:14Z
Hi, I encounter a similar problem.Have you guys fixed it? Any help?