Fintune whisper model returns exclamation marks
Hi, After finetuning whisper followed the blog (Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers), I find eval_wer, eval_cer score are bad.Because of that, I checked result of pred_str, about some pred_str setneces are just all exclamation marks. I checked also audio encoding part, but audio files are okay.Is there any reason the result of finetuning returns only exclamation marks?image2432×371 168 KBAnd these are my training arguments:“training_args = Seq2SeqTrainingArguments(output_dir=”./whisper-small_output2", # change to a repo name of your choiceper_device_train_batch_size=16,gradient_accumulation_steps=8, # increase by 2x for every 2x decrease in batch sizelearning_rate=1e-5,warmup_steps=50,max_steps=4000,gradient_checkpointing=False,fp16=True,tf32=True,dataloader_num_workers=4,evaluation_strategy=“steps”,per_device_eval_batch_size=8,predict_with_generate=True,generation_max_length=225,save_steps=50,eval_steps=50,logging_steps=1,report_to=[“tensorboard”,“wandb”], #, “wandb”load_best_model_at_end=True,metric_for_best_model=“wer”,greater_is_better=False,#push_to_hub=True,)"
2023-08-07T13:35:51Z
I would double-check the padding method you are using. Because sometimes, the padding function pads with 0, which is the token that represents the exclamation mark