Whisper fine-tuning and retaining timestamp decoding
Hi everyone,I am fine-tuning whisper models on some internal data, but I want whisper to retain its abilities besides just decoding texts.How do I fine-tune whisper yet retain its abilities of timestamp decoding and langdetectfurther more I am also trying to do multilingual fine-tuning where I take multiple langauges and fine-tune them together. how should I go about doing this ?
2023-07-10T17:58:03Z
gently pining@sanchit-gandhi
2024-05-06T11:51:01Z
@StephennFernandesGreat question, if@sanchit-gandhican help us about this matter it would be great
2024-05-12T14:11:36Z
i have the answer to it,it seems that if you do LoRA finetuning on whisper and prompt it to decode text with timestamps post funetuning you can still retain timestamps as LoRA fine-tuning doesn’t drift away whisper timestamp decoding capabilities while also generalizing well on the newly finetuned data.happy trainng!
2024-05-21T07:48:09Z
@StephennFernandesThanks man