What is cause and solution to Trainer error: cuda RuntimeError 711?
I’m trying to fine-tune DistilBert using Trainer, following the HuggingFace tutorial:https://huggingface.co/transformers/custom_datasets.html#fine-tuning-with-trainerWhen I try running the example, I get the following error:RuntimeError: cuda runtime error (711) : peer mapping resources exhausted at /pytorch/aten/src/THC/THCGeneral.cpp:139What does this error mean and how do I fix it?
2021-10-04T08:27:59Z
RylanSchaeffer:RuntimeError: cuda runtime error (711) : peer mapping resources exhaustedCan you run your code on CPU, to get a more informative error message?
2021-10-04T15:35:38Z
How do I specify in either theTrainingArgumentsor theTrainerto run on GPU?
2021-10-04T15:39:36Z
You can just run you script with"CUDA_VISIBLE_DEVICES=""before, to hide the GPUs
2021-10-04T16:57:16Z
I’m rerunning and no error emerges, but it’s as slow as mud