Finetuning whisper attention mask not set and canot be inferred
I was following@sanchit-gandhi's tutorial (https://huggingface.co/blog/fine-tune-whisper) but I got the following warning “The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input’sattention_maskto obtain reliable results.”.This warning was introduced in a recent transformers release and I was trying to understand if it can affect the finetuning process. If so, how can we avoid it?
2024-07-17T08:06:48Z
In my knowledge, I think the cause of the warning is theeos_token_id=pad_token_id=50257in the tokenizer, but it’s fine because we replaced thepad_token_idswith-100at the linelabels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)in the classDataCollatorSpeechSeq2SeqWithPadding.
2024-07-17T08:32:17Z
@duonguyenthat makes sense, thanks!
2024-07-20T18:52:14Z
I ran into the same error message when running this basic HF Whisper inference codes:https://huggingface.co/docs/transformers/en/model_doc/whisper#inferenceDoes anyone know how to fix it? It seems to me in Whisper’sdefault config:pad_token_id = 50256, eos_token_id = 50256. How can I make them not equal?
2024-07-20T21:55:45Z
This example forLongform transcriptiondoesn’t work out-of-the-box either:https://huggingface.co/docs/transformers/v4.42.0/en/model_doc/whisper#transformers.WhisperForConditionalGenerationIt runs into the same error message as above.I added a line:kwargs["attention_mask"] = attention_maskhere:https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/generation_whisper.py#L487And it worked and got reasonable ASR results.  I thought thetransformerslibrary changed some APIs at some point and Whisper didn’t pass in theattention_maskargument properly.Not sure what’s the optimal solution.