Mixed precision for bfloat16-pretrained models
Asbfloat16hardware support is becoming more available there is an emerging trend of training in bfloat16, which leads to the issue of not being able to finetune such models in mixed precision (or eval in fp16) - be it amp, apex or deepspeed/fairscale.Last week I spent some time sitting with the NaN issues reported in t5/mt5 (and pegasus apparently too), and I have been watching the activation values:[T5/MT5] resolve inf/nan under amp (mixed precision) by stas00 · Pull Request #10956 · huggingface/transformers · GitHuband studying the numerical qualities of bfloat16 vs bloat16:ml-ways/bfloat16-vs-float16-study.ipynb at master · stas00/ml-ways · GitHubSo my conclusion/understanding is this: sincebfloat16has no access to precision it basically compensates and trains itself to use huge numbers, so rather than having small activation values it operates in the1e5-1e10+ range which is beyond the 64k limitfloat16can handle and thus overflows (inf) which then immediately leads tonan(see mynbfor howinf/nancomes about).To make things worsebfloat16huge number range has huge gaps with no numbers in it:torch.tensor(283, dtype=torch.bfloat16)*10 # 2848 instead of 2830!so it trains to compensate for that handicap as well. And so whenfloat16comes around which has much smaller gaps it obviously won’t produce the same results. See my notebook to see the gaps demo’ed.Ideally there should be some plane transform that could take the weights trained in bfloat16 and convert those to the numerical domain of float16.  A naive approach could be to divide everything by ~100000 to shift to a different effective range . But because the training is non-linear I can’t see how this would be possible, other than via some DNN that was trained for such transform.As you can see from the PR some workarounds may work, but it’s hard to keep the numbers in check when the model wants to constantly operate in the range float16 wasn’t designed for. A user already reported NaNs after a 3h training with this PR, but hasn’t shared a way to reproduce yet.@sshleifersuggestedherethat perhaps finetuning with a penalty for large activations could do the trick. It’s unclear how much of such finetuning it’d take, since the need is to lower the weights by several orders of magnitude, so that the activations and accumulative math operations don’t break the 64K barrier.So currently t5/mt5/pegasus models are affected, but I’m sure there will be more emerging as new hardware supporting bfloat16 is quickly emerging so we will have to deal with that a lot more very soon I believe.Of course, if we wait long enough, the mixed precision will be moved to fp32/bf16 or even not be needed anymore.If perhaps some of you have experimented with such bf16 to fp16 finetuning and had good results please do share. It’s possible that if a solid approach is found then we will need to make a 2nd set of these models whose weights are finetuned for fp16.Thank you.
2021-04-06T19:44:58Z
And it looks like GPT-Neo just added itself to the group ofbfloat16-pretrained models.github.com/huggingface/transformersFP16 overflow with GPT-Neo when using sequence lengths of 2048.opened02:28AM - 06 Apr 21 UTCclosed03:10PM - 27 May 21 UTCLouisCastricato## Environment info

- `transformers` version: 4.5.0.dev0
- Platform: Linux-5….4.0-54-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0+cu111
- Tensorflow version (GPU?): N/A
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@stas00 

Models:

- GPT-Neo 1.3b

Library:

- deepspeed: @stas00

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Use GPT-Neo 1.3b with The Pile dataset and built in trainer. Artificial data also suffices. It does not matter what the data is, as long as the attention mask spans all 2048 tokens.
2. Enable FP16 and set max_length to 2048
3. Observe that all loses reported are NaN

Also reproducible using AMP or DeepSpeed. It seems like there is code to circumvent this outlined in the GPT-Neo implementation where q,k,v are casted to fp32 in the attention block. 

When the max_length is shorter (512) this overflow does not occur.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I expected no overflows.

## Aside

I'm reaching out on behalf of EleutherAI, Lysandre told us to create an issue about this.
2021-04-21T23:37:27Z
We started compiling a wiki of how different models were pre-trained, please add your knowledge there  - thanks!Model pre-training precision database: fp16, fp32, bf16🤗TransformersThere is an emerging need to know how a given model was pre-trained:  fp16, fp32, bf16. So one won’t try to use fp32-pretrained model in fp16 regime. And most recently we are bombarded with users attempting to use bf16-pretrained (bfloat16!) models under fp16, which is very problematic sincefp16 and bf16 numerical ranges don’t overlap too well. 
We arediscussing adding a new field tomodelsthat will tell users how it was trained. 
Some papers don’t disclose how a model was trained. So …