Using gradient_checkpointing=True in Trainer causes error with LLaMA
Hey,I am trying to fine tune llama using the transformers library. I noticed when I set gradient_checkpointing=True in the trainer args i get the following error:ExpectsBACKWARD_PREorBACKWARD_POSTstate but got HandleTrainingState.FORWARDHas anyone come across this before?@sguggerI have seen you respond to gradient_checkpointing questions before, so thought I would tag youScreenshot 2023-07-09 at 00.25.041010×1790 174 KB
2023-07-08T23:58:48Z
tsheasha:gradient_checkpointingThis happens when i have set use_reentrant=False in the call to torch.checkpoint, when i set use_reentrant=True, i get an error when torch.compile() runs on the model