Wav2vec2 xlsr nan train loss
Hi,I’m running intonantraining_losswhen training wav2vec2 xlsr with my custom dataset.Weird thing is that even thoughtraining_lossgoes tonan,eval_lossstill goes down, and error_rate (cerandwer) also goes down.I’ve experimented with lower learning_rate, but still getting similar behavior. I’m logging withwandb.My graphs look like the following:lr2400×1200 118 KBerror_rate2400×1200 139 KBloss2400×1200 122 KBThere’s no value fortrain/lossafter ~60 steps since it isnan, buteval/lossis still decreasing.Has anyone experienced similar behavior?
2021-06-14T04:22:09Z
I’ve let it train over the weekend, stillNANtrain loss, buteval lossand bothWERandCERcontinue to decrease