Transformers on GCP Training stuck on start
Hi everyone,I have a strange problem with running transformer models on Google Cloud VM instances.Once training should start, the program is stuck without an error message at***** Running training *****Num examples = 6838Num Epochs = 5Instantaneous batch size per device = 8Total train batch size (w. parallel, distributed & accumulation) = 8Gradient Accumulation steps = 1Total optimization steps = 4275Number of trainable parameters = 1094906990%|                                                  | 0/4275 [00:00<?, ?it/s]nvidia-smi shows that the GPU is idle, top shows the process idle.I thought I might have an error with my scripts, but even when I try one of the examples ( i.e.,Google Colab) the same error occurs.In Colab, the scripts run without a problem. The error occurs with “deep learning VM” image and barebone Linux VM. System, python packages, all up-to-date.I have seen some postings around the web with similar problems but no real solution. I wonder if someone figured it out?
2023-03-17T16:09:24Z
I am running into a similar issue (also on GCP). Even though I am using a differentscript. Training is stuck immediately.Did you find any solution?
2023-03-17T16:35:16Z
This was thesolutionfor me.
2023-03-22T18:41:41Z
As agademic wrote, uninstalling torch_xla did the trick for me.