ValueError fp16 lm_head.weight
I am trying to runrun_translation.pywith mt5-large and DeepSpeed enabled. I useds_config_zero3.jsonas the config file. However, when I try to run this, I get the following error:ValueError: fp16 is enabled but the following parameters have dtype that is not fp16: lm_head.weightIs there some config setting I’m missing that could help resolve this issue?
2021-10-24T15:58:29Z
Hey did you figure out to resolve this? I’d be interested to learn what you did.I ran the ASR examplehere, and it ran fine, but I noticed it had fp16 set to false. If I try to save memory by passing --fp16 at the command line or manually invoking fp16=True when calling TrainingArguments, I get the same error you report.