Training GPT2 on CPUs?
I want to make a pseudo-GPT3(GPT2 with more layers), but it’s too large for my 2*V100. And when I try to train on CPU, it gives me an error:Traceback (most recent call last):                  | 0/1260608 [00:00<?, ?it/s]File “examples/language-modeling/train.py”, line 336, inmain()File “examples/language-modeling/train.py”, line 300, in maintrainer.train(model_path=model_path)File “/home/ksjae/.local/lib/python3.7/site-packages/transformers/trainer.py”,line 741, in traintr_loss += self.training_step(model, inputs)File “/home/ksjae/.local/lib/python3.7/site-packages/transformers/trainer.py”,line 1055, in training_stepself.scaler.scale(loss).backward()File “/home/ksjae/.local/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py”, line 156, in scaleassert outputs.is_cudaAssertionError
2020-10-17T12:16:37Z
Seems like you tried to use AMP , which requires CUDA.  You should not use that flag when running in CPU mode.
2020-10-17T12:18:01Z
What’s AMP?
2020-10-17T12:33:59Z
Oh, and using block_size of 2048 causes this error:result = self.forward(*input, **kwargs)File “/home/ksjae/.local/lib/python3.7/site-packages/transformers/modeling_gpt2.py”, line 594, in forwardposition_embeds = self.wpe(position_ids)File “/home/ksjae/.local/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_implresult = self.forward(*input, **kwargs)File “/home/ksjae/.local/lib/python3.7/site-packages/torch/nn/modules/sparse.py”, line 126, in forwardself.norm_type, self.scale_grad_by_freq, self.sparse)File “/home/ksjae/.local/lib/python3.7/site-packages/torch/nn/functional.py”, line 1814, in embeddingreturn torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)IndexError: index out of range in self
2020-10-17T12:40:56Z
You are probably using--fp16, which triggers AMP (automatic mixed precision), which is not supported on CPU.