T5-xxl mlm distributed training?
Hi there,I want to train T5-XXL on a domain corpus (unsupervised, masked training). I’ve got 2 GPU á 40GB at hand, so I need to make use of Acclerator and Deepspeed.The example scriptrun_t5_mlm_flax.pyfrom the flax-examples does not support AcceleratorThe scriptrun_mlm_no_trainer.pyfrom the pytorch-examples supports Accelerator but not the T5’s particular training mode.Is it true that huggingface’s Flax support is in “maintenance mode”?thanksDaniel
2024-02-15T14:26:51Z
I think accelerator is designed for toch not flax.