Does fp16 training compromise accuracy?
Does fp16 training compromise accuracy?
2022-05-16T15:38:03Z
petarulev:training compromise accuracy?If not, why isn’t it enabled by default?
2022-05-17T21:49:45Z
Mixed precision training (fp16) is only possible on certain hardware and in some cases results intraining instabilitydepending on if the model was pre-trained using bfloat16.For older GPUs (before Volta/Turing), fp16 provides no speed up and will require more memory because both the fp16 values and fp32 values will be stored in memory. I’d recommend reading this:Performance and Scalability: How To Fit a Bigger Model and Train It Faster