What to do for non-finite warning in `clip_grad_norm`?
I started to see this warning for a language model trainingFutureWarning: Non-finite norm encountered in torch.nn.utils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.Is this an indicator that my model is not working well? And if so, is there any recommendation on what to change? Thanks!
2021-07-26T22:10:35Z
Any insights on this?
2021-09-10T22:31:58Z
@sguggerCan you please provide some insight ? I get this warning even before the training starts withTrainer.
2021-09-13T07:22:39Z
I have never encountered that warning, so will look into that. It looks like some future change in PyTorch is coming.