How to debug NaN output of a logits in training

This has been a recurring problem because apparently I made my own id2labels.json, you can look at it if it adds more context in here. I did try “Getting all possible labels by grouping it after each inference” in this question for better context and the program actually confused at what to segment.

I tried making it 0 and 1 like what I had today and actually get probably a false positive due to small epoch on first training on here. So how is it really to make an id2label from a pretrained model id2label?

Thanks for the example, will try to see how to implement that in my code.

Btw if I just let the model training running, does the model actually broke or I’m just missing some stats I could probably circumvent by it’s products like maybe creating new masks of the new model and count the MIoU manually?

1 Like