Adding prompt / context to Whisper with Huggingface Transformers
The Whisper model, has the possibility of a prompt or adding the previous text to the current transcription task. This helps in case of transcribing long file chunk after chunk.During training it should “mask out the training loss over the previous context text, and train themodel to predict all other tokens”.arxiv.org2212.04356.pdf985.00 KBI’m wondering if HF has implemented that and how well does it helps with the accuracy.There is also this issue in Huggingface Transformers:github.com/huggingface/transformersWhisper doesn't compute positional embeddings properly when given batches of prompt tokensopened05:34PM - 06 Dec 22 UTCclosed03:02PM - 14 Jan 23 UTCandyehrenberg### System Info

v4.25.1 on M1 Mac with python 3.8

### Who can help?

@sanchit-…gandhi @patrickvonplaten @anton-l 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When we want to run Whisper generation for a batch of samples with different prompt lengths (prefix tokens given to the decoder), positional embeddings for the decoder are improperly computed. It assumes all sequences have the same `past_key_values_length`, but this is not true in general.

Scenario:
`decoder_input_ids = [50361, 45431,  2584, 28682, 13, 50258, 50257, 50257]`
(`"<|startofprev|>Something completely irrelevant.<|startoftranscript|><|pad|><|pad|>"`)

`model.generate(input_features, decoder_input_ids=decoder_input_ids, decoder_attention_mask=decoder_attention_mask)` will not give the correct output because at the beginning of decoding, the pad tokens won't be taken into account that the positional embedding will be off.

### Expected behavior

Instead of tracking `past_key_values_length`, it should use the attention mask to compute position ids. The current implementation is more based off of encoder-decoder architectures that would never do decoder prompting, but it should take more inspiration from decoder-only models to handle prompting. This is done for the Flax implementation in #20479I’m wondering if it is fixed and stable.There it seems that the way of doing it with HF would be is something like so:prev_before = '<|startofprev|>'
current_before = "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>"
current_after = "<|endoftext|>"

prev_text = 'Hello and'
current_text = 'welcome. My name is...'

prompt_and_text = prev_before + prev_text + current_before + current_text + current_after
prompt_and_text<|startofprev|>Hello and<|startoftranscript|><|en|><|transcribe|><|notimestamps|>welcome. My name is…<|endoftext|>prompt_and_text_tokens = _tokenizer.encode(prompt_and_text, add_special_tokens=False)
_tokenizer.decode(prompt_and_text_tokens)<|startofprev|>Hello and<|startoftranscript|><|en|><|transcribe|><|notimestamps|>welcome. My name is…<|endoftext|>
2023-03-22T20:54:05Z
@SamuelAzrandid you manage to get this working? I’d like to have the exact same functionality.
2023-03-24T13:35:20Z
Any news on this?
2023-03-24T15:28:17Z
I did not see Huggingface supporting it or providing an easy way to do it, like in the open_ai library, but you can hack around or use lower-level components in HF to give context or prompt in inference time. In training time, training on prompts is much more complicated. I hope that someone from HF can address this issue.The benefits of using prompts are beyond just giving the text of the previous speech segments. You can also provide prompts to control the style of the transcript and add relevant domain-specific or industry-specific terms to the context so the transcription process will be more likely to use those terms in the final transcript.
2023-03-24T19:29:55Z
Great! Thanks! That’s what I will end up doing
2023-07-11T09:50:46Z
Related discussion:Is it possible to do model adaptation? · openai/whisper · Discussion #66 · GitHubYou can also use logit biasing to increase the probability of certain domain specific words being predicted.
2024-04-30T08:10:19Z
Any update on support for this in hf libraries?