FP-16 training producing nans on t5-large/flan-t5-xl
This was an issue a while back but seems to have resurfaced -T5 fp16 issue is fixedI have tested the exact following code ont5-smallandt5-baseand they work fine. However, when usingt5-largeand/orflan-t5-xl, the model produces nan outputs. This is solely a result of using half precision (ignore the multiple GPUs, strategy etc, I have tested with every other variation):trainer = pl.Trainer(
    precision="16",
    accelerator='gpu',
    strategy='auto',
    devices=4,)I am usingtransformers == 4.28.1andlightning == 2.0.0Any ideas/help appreciatedThanks!