Does transformers 3.5.1 support auto mixed precision training？
I’m using a GPTLMHead model in pytorch.Is it possible , i add autocast() in the forward function in GPTLMHead and change the training process followed theAutomatic Mixed Precision — PyTorch Tutorials 1.8.1+cu102 documentation
2021-05-26T11:51:19Z
Yes, it’s possible.
2021-06-01T02:48:09Z
I train the gpt model using V100 with mixed precision training. But it happens that in the inference time, the generated token is mess with the autocast() open.when i turn off it, everything is ok.