Trainer.train() is stuck
Hi,I’m training roberta-base using HF Trainer, but it’s stuck at the starting itself. Here’s my code -train_dataset[0]
{'input_ids': tensor([  0, 100, 657,  ...,   1,   1,   1]),
 'attention_mask': tensor([1, 1, 1,  ..., 0, 0, 0]),
 'labels': tensor(0)}

val_dataset[0]
{'input_ids': tensor([    0, 11094,    14,  ...,     1,     1,     1]),
 'attention_mask': tensor([1, 1, 1,  ..., 0, 0, 0]),
 'labels': tensor(0)}

## simple test
model(train_dataset[:2]['input_ids'], attention_mask = train_dataset[:2]['attention_mask'], labels=train_dataset[:2]['labels'])
SequenceClassifierOutput(loss=tensor(0.6995, grad_fn=<NllLossBackward>), logits=tensor([[ 0.0438, -0.1893],
        [ 0.0530, -0.1786]], grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)

train_args = transformers.TrainingArguments(
             output_dir='test_1',
             overwrite_output_dir=True,
             evaluation_strategy="epoch",
             per_device_train_batch_size=8,
             per_device_eval_batch_size=8,
             learning_rate=3e-5,
             weight_decay=0.01,
             num_train_epochs=2,
             load_best_model_at_end=True,
             )

trainer = transformers.Trainer(
             model=model,
             args=train_args,
             train_dataset=train_dataset,
             eval_dataset=val_dataset,
             tokenizer=tok,
             )

trainer.train()I saw memory consumption and it is stuck at -+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:62:00.0 Off |                    0 |
| N/A   49C    P0    60W / 300W |   1756MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
| N/A   50C    P0    61W / 300W |   1376MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+Plz let me know how to proceed further…
2021-05-26T11:37:17Z
any progress since then?
2022-02-01T15:10:25Z
check if works with num_worker=0 if you are using windows.It worked for me. here is referencePyTorch Forums – 2 May 19Cannot enumerate over Dataloader objectIs your code running fine using num_workers=0?  If you are using multiple workers on Windows, you should add the if-clause protection as described in the Windows FAQ.
2022-08-31T11:10:14Z
I am also having the same problem. When training the trainerimage1216×494 28.7 KBI have used the urdu dataset of mozilla-foundation/common_voice_7.0. The model which I have used is “facebook/wav2vec2-xls-r-300m” and the libraries and transformers are!pip install datasets ==2.4.0!pip install transformers ==4.21.2!pip install torchaudio = 0.11!pip install jiwer"If anyone could help me, I would be very thankful.
2023-01-11T10:59:04Z
I ran into the same problem. I am using GCP and when I create notebook using the PyTorch, when I launch the trainer there is no progression. I manage to solve the problem by creating a choosing to create an environment without torch and by installing torch using the following line:pip install torch==1.13.1+cu117  -f https://download.pytorch.org/whl/torch_stable.htmlI hope it helps.
2023-05-01T04:36:14Z
I ran into a similar problem while I was using GCP.  To resolved the problem I edit my instance to have more memory from 4GB to 30GB.  I ran out of system memory and the only reason I realized it was because other tasks I was running at that time (non torch/GPU related) threw error.  The Trainer.train() returns no error and just hung for hours (I first thought I setup my GPU instance wrong and didn’t have the correct driver)