Does using FP16 help accelerate generation? (HuggingFace BART)
I also this question in StackOverflow, but couldn’t get a response yet (pytorch - Does using FP16 help accelerate generation? (HuggingFace BART) - Stack Overflow).I follow the guide below to use FP16 in PyTorch.pytorch.orgPyTorchAn open source machine learning framework that accelerates the path from research prototyping to production deployment.Basically, I’m using BART in HuggingFace for generationDuring the training phase, I’m able to get 2x speedup and less GPU memory consumptionBut.I found out there is no speedup when I callmodel.generateundertorch.cuda.amp.autocast().with torch.cuda.amp.autocast():
   model.generate(...)When I save the model by:model.save_pretrained("model_folder")the size does not decrease to half. But I have to callmodel.half()before saving in order to make the model half size.Thus, my questions:Is the issue in1.expected or there should be something I did wrong?Is the operation I did in2.proper?
2020-09-29T09:06:25Z
To use the model for inference infp16you should callmodel.half()after loading it.Note that callinghalfputs all models weights infp16, but in mixed precision training some parts are still kept infp32for stability (likesoftmaxlayers), so it might be a better idea to useampin01opt mode instead of callinghalf.I’m not sure iftorch.cuda.amp.autocast()helps in inference, better to ask this on pytorch forum
2020-09-30T03:11:09Z
I actually tried usingapexin 01 mode but it does not help the generation speed as well. But it did help improve the training speed of seq2seq bart model. Is it also in your case?