Very Slow Fine Tuning Performance for Speech?
I am trying to fine tune the Facebook 60K hour wav2vec2 model as described in Patrick von Platen’s  article:Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 TransformersI have about 200 hours of speech data. I am using 3 RTX8000 GPUs on a 48 Core Lenovo SR670 with 369 GB of memory (on our NYU compute cluster).It seems to be using all 3 GPUs (utilization is high)  but seems as slow as molasses. If I read the output right it looks like it will take over a week to fine tune the model with 30 epochs.Does this sound correct?ThanksMichael Picheny
2023-08-14T18:06:32Z
are you found out the problem?
2023-08-14T18:35:13Z
I just ran it for a week.
2023-08-14T20:30:32Z
dam, i’m working with 1200 hour of Bengal speech and use Lora and PEFT for finetune the model but time and resource usage is equivalent to full finetune of the model.