Does anyone have working code for training T5-11B on multi-gpu?
I have tried so many variations of completely different code and I can’t get it working.  I can train any of the smaller t5 models, but once it requires multi-gpu, I can’t get it to work. Tried deepspeed, accelerate, and solutions without using either of those.So, my question is, does anyone have some code that runs t5-11b for fine tuning? (or know where some is)
2023-01-18T13:04:20Z
I’d check these threads:[DeepSpeed] [success] trained t5-11b on 1x 40GB gpu · Issue #9996 · huggingface/transformers · GitHubhttps://github.com/huggingface/transformers/issues/14531
2023-01-18T13:15:49Z
Thanks, I have read through those on previous attempts, but didn’t get them working.
2023-03-30T10:20:22Z
Hi, I am also trying to train T5-11b on multi-gpu. Have you succeeded?
2023-03-30T10:47:39Z
No, I didn’t get it to work properly.