TPU slow finetuning T5-base
Hi,I am trying to train a T5-base on Colab with the TPU. I am using the official code to perform a fine-tuning on the T5-base (with my dataset), but the training with TPU is extremely slow! I’m using the officalcode.I am attaching the colab code with the various libraries I have installed:notebook.Also, if I try to increase the batch size as >= 64, I get a memory error, as there seems to be only about 8 Gb available.Can someone help me? Thank you!
2022-02-16T13:35:28Z
Could you please post how you run this code? I mean, do you usexla_spawn.py, are you running it inline on a notebook?
2022-02-16T13:37:51Z
@finiteautomataI’m running it on Google colab, online with Google Colab PRO
2022-02-16T13:54:53Z
@GenVsorry, I didn’t see yournotebook link. You are missing thexla_spawn.pypart, that is, the code that makes your code run in a parallel fashion. You should add this:python xla_spawn.py --num_cores 8  t5.py \
    --model_name_or_path="t5-base" \
    --do_train \
    --do_eval \etc etc
2022-02-16T14:05:46Z
@finiteautomataso I needthiscode and runpython xla_spawn.py --num_cores 8  t5.py \  and all other args of t5.py ?
2022-02-16T14:08:45Z
Exactly that. Try it and tell what happens
2022-02-16T15:36:20Z
@finiteautomataI don’t know if it’s doing all well, I have this prints:1- Running tokenizer on train dataset:   0% 0/30 [00:00<?, ?ba/s]WARNING:t5:Process rank: -1, device: xla:0, n_gpu: 0distributed training: False, 16-bits training: FalseWhere xla:0 e not 1, but maybe it’s for the tokenizer run2- huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks…To disable this warning, you can either:- Avoid usingtokenizersbefore the fork if possible- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)Now it’s in finetuning on step 1.
2022-02-16T16:25:42Z
@finiteautomataI see that at start xla = 1, then xla = 0. Furthermore, the training seems to take a long time, about 90 hours (on gpu they become less than 2); maybe somehow it uses the CPU?
2022-03-02T23:29:28Z
@GenVI have the same problem when using the TPU in Colab (I have Google Colab Pro +). I was not using thexla_spawn.py, I gave it a try and interestingly, the first time I did run my script using thexla_spawn.pyit made my training faster, however, after terminating my node and reconnecting to the TPU, I cannot make it work again and even using thexla_spawn.pythe training is very slow (so it was kind of random and I can’t reproduce).Did you figure something out?
2022-03-03T03:10:37Z
For folks who are still struggling, I think I found one potential reason for why training on TPU is slow, lookhere(I set padding toTruein my tokenizer and I’m already seeing a speedup in my TPU training, basically it looks like it didn’t have anything to do with my torch xla installation)
2022-03-03T14:19:52Z
Thank you@phosseinifor your answer! Yes I had the same issue with the randomness. I have read the discussion and it is interesting, so it will be forpadding = True(I was using False if I’m not mistaken).Also I have your same question (if you just need to putpadding = False), I wait for answer in the other thread.
2022-03-08T09:38:21Z
@phosseiniI tried withpad_to_max_length=Trueand it’s working fine.
2022-06-16T08:52:22Z
Did you guys notice speedups vs GPU training?@GenVI have paid access to A100 GPUs but for side research tasks I’d like to use TPUs in case something works out…
2022-06-17T07:36:27Z
@deathcrushThe tpu is much faster VS gpu training with a A100, P100, V100 .