Run_mlm.py cuda error memory after resuming a training
Hello,I am trying to pretrain an XLM Roberta model, but I got some issue. I have trained the model for few thousand steps and get the checkpoint. Then I wanted to continue the pre-training from the checkpoint, but got a memory error “CUDA error memory”  after few steps. I wonder if there is a leak of something like that ? The memory used during the 1st pretraining used around 15.X GB / 16.2 GB so I quite don’t understand what’s going on.
2021-04-17T04:09:58Z
I’m running into the same issue but with the mBART model. For some reason, running training from scratch with theSeq2SeqTrainerworks just fine, but resuming from checkpoint exceeds the memory limit, and produces a CUDA ‘out of memory’ error.I think it might be related tothisissue on the GitHub repository.@sshleiferI think this is another issue with training large models, as we discussedherealthough this just seems to be a bug in the trainer.
2021-04-20T12:09:48Z
This fix by@sguggerin arecent commitresolved the issue for me. The fix is not in the official release yet, but you can install the bleeding edge version of the Transformers library from source usingpip install git+https://github.com/huggingface/transformerswhich pulls it from GitHub.
2021-04-21T14:34:21Z
having the same issue here with GPT-2 large, gonna try what@IamAdiSrisuggested
2021-04-21T15:10:48Z
Can confirm that installing the bleeding edge version of transformers as suggested by@IamAdiSrifixed the issue.