T5 fp16 issue is fixed
We have just fixed the T5 fp16 issue for some of the T5 models!(Announcing it here, since lots of  users were facing  this issue and T5 is one most widely used model in the library)TL;DR:Previously, there was an issue when usingT5models infp16; it was producingnanlossandlogits. Now on the master, this issue is fixed for the following T5 models and versions. Now you should be able to train and use these models for inference infp16and see a decent speed-up!T5v1:t5-small,t5-base,t5-largeT5v1_1:google/t5-v1_1-small,google/t5-v1_1-baseMT5:google/mt5-small,google/mt5-baseFor those of you who are interested, here’s a description of what was causingnanloss and how it is fixed.t5-smallwas the only T5 model that was working in fp16. The rest of the models producenanloss/logits.for all the models and versions (v1, v1.1, mT5), at some point, we getinfvalues inhidden_statesafter applying the final linear layer (wo) inT5DenseReluDenseandT5DenseGatedGeluDense.github.comhuggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L248-L278class T5DenseReluDense(nn.Module):def __init__(self, config):super().__init__()self.wi = nn.Linear(config.d_model, config.d_ff, bias=False)self.wo = nn.Linear(config.d_ff, config.d_model, bias=False)self.dropout = nn.Dropout(config.dropout_rate)def forward(self, hidden_states):hidden_states = self.wi(hidden_states)hidden_states = F.relu(hidden_states)hidden_states = self.dropout(hidden_states)hidden_states = self.wo(hidden_states)return hidden_statesclass T5DenseGatedGeluDense(nn.Module):def __init__(self, config):super().__init__()self.wi_0 = nn.Linear(config.d_model, config.d_ff, bias=False)self.wi_1 = nn.Linear(config.d_model, config.d_ff, bias=False)This file has been truncated.show originalwhich results innanvalues inT5LayerNorm.Also fort5-large,t5-v1_1-base,t5-v1_1-large, there areinfvalues in the output ofT5LayerSelfAttentionandT5LayerCrossAttention, specifically where we add the attn output with thehidden_statesgithub.comhuggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L548normed_hidden_states = self.layer_norm(hidden_states)attention_output = self.SelfAttention(normed_hidden_states,mask=attention_mask,position_bias=position_bias,head_mask=head_mask,past_key_value=past_key_value,use_cache=use_cache,output_attentions=output_attentions,)hidden_states = hidden_states + self.dropout(attention_output[0])outputs = (hidden_states,) + attention_output[1:]  # add attentions if we output themreturn outputsclass T5LayerCrossAttention(nn.Module):def __init__(self, config):super().__init__()self.EncDecAttention = T5Attention(config, has_relative_attention_bias=False)self.layer_norm = T5LayerNorm(config.d_model, eps=config.layer_norm_epsilon)self.dropout = nn.Dropout(config.dropout_rate)github.comhuggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L584normed_hidden_states,mask=attention_mask,key_value_states=key_value_states,position_bias=position_bias,head_mask=head_mask,past_key_value=past_key_value,use_cache=use_cache,query_length=query_length,output_attentions=output_attentions,)layer_output = hidden_states + self.dropout(attention_output[0])outputs = (layer_output,) + attention_output[1:]  # add attentions if we output themreturn outputsclass T5Block(nn.Module):def __init__(self, config, has_relative_attention_bias=False):super().__init__()self.is_decoder = config.is_decoderself.layer = nn.ModuleList()self.layer.append(T5LayerSelfAttention(config, has_relative_attention_bias=has_relative_attention_bias))This happens during both training and inference, to reproducefixTo avoidinfvalues we could clamp thehidden_statesto the max values for the current data type if there areinfin it. i.eif torch.isinf(hidden_states).any():
    clamp_value = torch.finfo(hidden_states.dtype).max - 1000
    hidden_states = torch.clamp(hidden_states, min=-clamp_value, max=clamp_value)we need to add this after self attn, cross-attn, and the feed-forward layer which is where theinfvalues occur. This works for bothapexandampTo verify this fix, trainedt5-base,t5-v1_1-baseandt5-v1_1-smalloncnn/dmfor 10k steps (1.11 epochs)Here’s the training command, to run this  navigate toexamples/seq2seqdir, follow the instructions in the readme to downloadcnn_dmand dataset, and then run the following commandexport M=google/t5-v1_1-base
export OUT_DIR=t5-v1_1-base-cnn-fp16
export DATA_DIR=cnn_dm

python finetune_trainer.py \
    --model_name_or_path $M \
    --data_dir $DATA_DIR \
    --output_dir $OUT_DIR --overwrite_output_dir \
    --max_steps=10000 \
    --gradient_accumulation_steps=8 \
    --learning_rate=1e-4 \
    --per_device_train_batch_size=4 \
    --n_val 500 \
    --max_target_length=56 --val_max_target_length=128 \
    --fp16 --fp16_backend apex \
    --do_train --do_eval --evaluation_strategy steps \
    --logging_steps=100 --logging_first_step --eval_steps=2500 --save_steps=2500 --save_total_limit=2 \
    --sortish_sampler \for evaluationpython run_eval.py \
    t5-v1_1-base-cnn-fp16  cnn_dm/test.source hypothesis.txt \
    --reference_path cnn_dm/test.target \
    --score_path metrics.json \
    --device cuda:0 \
    --prefix summarize: \
    --bs 16 \
    --fp16 \and got the following metrics (ROUGE2)fort5-base: 19.2804fort5-v1.1-base: 18.4316(note that the score fort5-baseis more because it’s already pre-trained oncnn/dm)To compare this, evaluated the pre-trainedt5-basein bothfp32andfp16, which gave the following resultsfp16: 18.3681fp32: 18.394So the results are close enough.To verify the fix fort5-large, evaluated the pre-trainedt5-largeinfp32andfp16(use the same command above to evaluatet5-large)  and got the following resultsfp16: 19.2734fp32: 19.2342Surprisingly, rouge2 is slightly better infp16.So with the above fix, the following model types now work infp16(opt level01), and give descent speed-upT5v1:t5-small,t5-base,t5-largeT5v1_1:google/t5-v1_1-small,google/t5-v1_1-baseMT5:google/mt5-small,google/mt5-baseOne interesting observation,For inference, thet5-basefine-tuned withfp16and evaluated infp32is faster (~1.31x) than pre-trainedt5-baseevaluated infp16.  See thiscolab
2021-01-12T22:17:11Z
Nice fix!The speed discrepancy might be because of different length generations.
2021-01-26T16:44:17Z
Very cool! Works well on T4 as well.Any guesses why the inference time for fp16 doesn’t seem to be noticeably faster? Saw only a small difference in the shared colab, and seeing similar behavior locally…
2021-01-29T09:00:02Z
Hey@aaronchavezas explained in this issueImprove PyTorch examples for FP16 · Issue #9752 · huggingface/transformers · GitHubTo get the full speed-up of FP16 training, every tensor passed through the model should have all its dimensions be a multiple of 8.
2021-01-29T15:21:18Z
Thanks for replying! I was wondering about inference in particular (via generate())…is it the same situation, there? I am seeing approximately the same times for generation with fp16 and fp32.
2021-03-21T14:21:33Z
HiThis is not fixed as far as I see, could you have a look here please:github.com/huggingface/transformersgetting nans with t5-large + fixopened08:52AM - 21 Mar 21 UTCclosed03:03PM - 23 Jul 21 UTCyuvalkirstain## Environment info

- `transformers` version: 4.5.0.dev0
- Platform: Linux-4….15.0-65-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patil-suraj @patrickvonplaten 

## Information

Model I am using (Bert, XLNet ...): t5-large

The problem arises when using:
* [ ] my own modified scripts: run_seq2seq with minor modifications (attached)


The tasks I am working on is:
* [ ] my own task or dataset: Closed-Book Open Domain QA

## To reproduce

Steps to reproduce the behavior (the fix I'm suggesting is very simple, so perhaps there is no reason to reproduce):
1. unzip the attached zip (below).
2. run 
```bash
python run_seq2seq.py --model_name_or_path=t5-large
--do_train
--do_eval
--task=qa
--train_file=data/PAQ.filtered.regular.16000.json
--validation_file=data/PAQ.filtered.regular.16000.json
--output_dir=results/5e-5-t5-large-4096000-128-140-1792000-0.1-regular-true-4
--overwrite_output_dir
--per_device_train_batch_size=1
--per_device_eval_batch_size=128
--predict_with_generate
--fp16
--max_steps=1000
--evaluation_strategy=steps
--text_column=question
--summary_column=answer
--save_total_limit=5
--cache_dir=../.cache
--save_steps=500000
--learning_rate=5e-5
--eval_steps=96000
--warmup_steps=100
--run_name=5e-5-t5-large-4096000-128-140-1792000-0.1-regular-true-4
--dropout_rate=0.1
--gradient_accumulation_steps=1
--logging_steps=1
```

## Expected behavior
Training without nans. 

## Possible fix
I debugged and saw that we get nans at the `modeling_t5.py` script in line 241 
```python
hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
```
By modifing this line to:
```python
clamp_value = torch.finfo(hidden_states.dtype).max - 1000
hidden_states = torch.clamp(hidden_states, min=-clamp_value, max=clamp_value) * torch.rsqrt(variance + self.variance_epsilon)
```
It seems to be solved.

BTW it happens in the last layers (this might explain why it wasn't caught in [this fix](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139))

[seq2seq.zip](https://github.com/huggingface/transformers/files/6177063/seq2seq.zip)github.com/huggingface/transformersmt5 getting nans with fp16opened06:44AM - 20 Mar 21 UTCclosed03:18PM - 07 Jun 21 UTCdorost1234## Environment info
<!-- You can run the command `transformers-cli env` and cop…y-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.8 
- Tensorflow version (GPU?): - 
- Using GPU in script?: - 
- Using distributed or parallel set-up in script?: - 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

 t5: @patrickvonplaten, @patil-suraj

## Information

I am using mt5-small model:
* the problem arises when using fp16 with mt5

The tasks I am working on is:
* translation

## To reproduce

Steps to reproduce the behavior:

`python run_translation.py     --model_name_or_path google/mt5-small     --do_train     --do_eval     --source_lang en     --target_lang ro     --dataset_name wmt16     --dataset_config_name ro-en     --output_dir  test/tst-translation     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir     --predict_with_generate --max_train_samples 100 --fp16`

outputs:
```
***** eval metrics *****
  epoch                     =     3.0
  eval_bleu                 =  0.0039
  eval_gen_len              =    2.95
  eval_loss                 =     nan
  eval_mem_cpu_alloc_delta  =     4MB
  eval_mem_cpu_peaked_delta =     5MB
  eval_mem_gpu_alloc_delta  =     0MB
  eval_mem_gpu_peaked_delta =  1080MB
  eval_runtime              = 72.1865
  eval_samples              =    1999
  eval_samples_per_second   =  27.692
```


## Expected behavior

being able to use fp16 with mt5 models. Thank you very much for your help, this is really crucial for me to be able to run these models with fp16 to be able to fit more data into old GPUs I have access to and I appreciate a lot your help.I really need these models and I appreciate having a look into it.thanks a lot
2021-03-21T14:22:29Z
@valhalla@patrickvonplaten
2021-03-21T15:58:10Z
Hi Dara,Can you specify which model you’re trying to use?
2021-03-21T17:45:06Z
Hi thereI use mt5-small, more info provided in the link to the issues me and another users opened thanks
2021-07-24T05:11:18Z
@valhallaI tried “finetune_trainer.py” you mentioned command.I got this error.07/23/2021 06:26:38 - INFO - __main__ - *** Train ***
/usr/local/lib/python3.7/dist-packages/transformers/trainer.py:1026: FutureWarning: `model_path` is deprecated and will be removed in a future version. Use `resume_from_checkpoint` instead.
  FutureWarning,
Traceback (most recent call last):
  File "/content/gdrive/MyDrive/Colab Notebooks/generation/transformers/examples/legacy/seq2seq/finetune_trainer.py", line 367, in <module>
    main()
  File "/content/gdrive/MyDrive/Colab Notebooks/generation/transformers/examples/legacy/seq2seq/finetune_trainer.py", line 305, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1138, in train
    self.create_optimizer_and_scheduler(num_training_steps=max_steps)
  File "/content/gdrive/MyDrive/Colab Notebooks/generation/transformers/examples/legacy/seq2seq/seq2seq_trainer.py", line 118, in create_optimizer_and_scheduler
    if self.sharded_dpp:
AttributeError: 'Seq2SeqTrainer' object has no attribute 'sharded_dpp'I have no idea how to fix this issue. How can I fix this?
2022-01-20T07:29:15Z
Hi@valhallaThe fix seems not to be applied on the MT5 model:Nan losses occur when fine-tuning the model with the latest (stable) version of transformers.MT5 works well with fp16.Do you see the same behavior?
2023-02-01T17:22:06Z
I had the same issue and I changed it to fp16=False and it worked. Although the loss became a non-nan value, it caused the Out Of Memory error frequently. Impossible to train more. I tested many things and finally I tried this:fp16_full_eval=True instead of fp16l=True. It solved the loss problem but at the end ROUGE metrics became all zero. How can it compute the loss but not the metrics?
2023-02-03T11:36:53Z
Are the changes already merged?I am unable to fine-tune mT5 with fp16.
2023-04-05T18:38:13Z
@valhallaSo where is this fix now? I do not get where I should place code? Is it in anew version of transformers?
2023-04-27T22:37:35Z
I am getting same issue withgoogle/flan-t5-smallmodel withfp16=True. Is the error reported for this model too?
2023-07-01T14:19:20Z
I am getting the same issue withSalesforce/blip2-flan-t5-xlwhich usesgoogle/flan-t5-xlwithamp.I am usingtransformers==4.29.2.I gotinfvalues inhidden_statesafter applying the final linear layer (wo) inT5DenseGatedActDense.I found this by using PyCharm debug mode, as shown in the screenshot below:屏幕截图 2023-07-01 2155531563×1500 255 KBBy the way, I didn’t find the fixing code which clamps thehidden_statesto the max values described infixin theT5DenseGatedActDenseclass.
2023-07-01T15:14:54Z
I am usingtransformers==4.29.2.I may have found a logical flaw inT5Blockas shown in below:BlockquoteApply Feed Forward layerhidden_states = self.layer-1clamp inf values to enable fp16 trainingif hidden_states.dtype == torch.float16:clamp_value = torch.where(torch.isinf(hidden_states).any(),torch.finfo(hidden_states.dtype).max - 1000,torch.finfo(hidden_states.dtype).max,)hidden_states = torch.clamp(hidden_states, min=-clamp_value, max=clamp_value)BlockquoteApparently the fixing code which clamps thehidden_statesto the max values only works whenhidden_statesis fp16. It may works fine when the whole model is fp16. But I am trying to do mixed precision training usingtorch.amp, so thehidden_statesreturned byT5LayerFFis fp32 because the last operation inT5LayerFFis the adding op, a.k.a the residual connection, which can’t autocast tofloat16according tohttps://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float16.The screenshot below also proved this theory:屏幕截图 2023-07-01 2250031761×977 88.5 KB
2024-05-27T05:00:53Z
3 year later, it seems t5-large still doesn’t support fp16. Anyone found a solution?
2024-06-20T19:10:10Z
lexandstuff:3 year later, it seems t5-large still doesn’t support fp16. Anyone found a solution?You should use bf16 instead of fp16 for T5 models