text
Why does returning timestamps help Whisper reduce hallucinations? 🧐
Why does returning timestamps help Whisper reduce hallucinations? 🧐
new_line
new_line
text
Empirically, most practitioners have found that setting 
Empirically, most practitioners have found that setting 
inline_code
`return_timestamps=True`
return_timestamps=True
text
 helps reduce hallucinations, particularly when doing long-form evaluation with Transformers’ “chunked” algorithm. 
 helps reduce hallucinations, particularly when doing long-form evaluation with Transformers’ “chunked” algorithm. 
new_line
new_line
text
But why does this work?..
But why does this work?..
new_line
new_line
text
My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
new_line
code_fence
```markdown
The cat sat on the on the on the mat.
```
The cat sat on the on the on the mat.
markdown
new_line
text
Where we have a repeated hallucination for “on the”. If we ask the model to predict timestamps, then the “on the” has to contribute to the overall segment-level timing, e.g.:
Where we have a repeated hallucination for “on the”. If we ask the model to predict timestamps, then the “on the” has to contribute to the overall segment-level timing, e.g.:
new_line
code_fence
```markdown
<|0.00|> The cat sat on the on the on the mat.<|5.02|>
```
<|0.00|> The cat sat on the on the on the mat.<|5.02|>
markdown
new_line
text
However, it’s impossible to fit 3 copies of “on the” within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
However, it’s impossible to fit 3 copies of “on the” within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
new_line
code_fence
```markdown
<|0.00|> The cat sat on the mat.<|5.02|>
```
<|0.00|> The cat sat on the mat.<|5.02|>
markdown
new_line
text
In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper 
In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper 
resource
https://huggingface.co/papers/2212.04356
paper
2212.04356
https://huggingface.co/papers/2212.04356
Robust Speech Recognition via Large-Scale Weak Supervision (2212.04356)
text
 → it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).
 → it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).
new_line
new_line
text
Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?
Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?
Why does returning timestamps help Whisper reduce hallucinations? 🧐

Empirically, most practitioners have found that setting `return_timestamps=True` helps reduce hallucinations, particularly when doing long-form evaluation with Transformers’ “chunked” algorithm. 

But why does this work?..

My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
```markdown
The cat sat on the on the on the mat.
```
Where we have a repeated hallucination for “on the”. If we ask the model to predict timestamps, then the “on the” has to contribute to the overall segment-level timing, e.g.:
```markdown
<|0.00|> The cat sat on the on the on the mat.<|5.02|>
```
However, it’s impossible to fit 3 copies of “on the” within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
```markdown
<|0.00|> The cat sat on the mat.<|5.02|>
```
In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper https://huggingface.co/papers/2212.04356 → it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).

Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?
❤️
clem abellion vladbogo daedaluswave osanseviero samusenps Csplk DRRRRDFFD on1onmangoes sanchit-gandhi jayzhuo michaelsymph EBalykov Mutturaj DattaKiran OjciecTadeusz ericlaycock dkakaie WIldOni ayoalfonso dzlandis peterciank
👍
emirvmendoza oyemade EBalykov Mutturaj
🤗
nss-ysasaki
👀
Wok
➕
Mutturaj
/avatars/13fbba68d1d4d29afa0d2ac1a24faa28.svg
Raj Gothi
RajGothi
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1653243468328-61f91cf54a8e5a275b2b3e7c.jpeg
Sanchit Gandhi
sanchit-gandhi
user
/avatars/c83664b8b11a1e72a41866e98e377426.svg
Sai
Chandrahema
user
https://cdn-avatars.huggingface.co/v1/production/uploads/670203bca5a482f20ad7aa47/Dec9xTwFkCeSLti_9tYPO.png
Danielle
Sin2pi
user
/avatars/ec2b0fa1dca861fdf3e41b33cf27bbc3.svg
Regent Yung
regentyung
user
https://cdn-avatars.huggingface.co/v1/production/uploads/64cdcbef244a7de1132247f4/dEezgF4LW9lqxAgvmc2oH.jpeg
Wadi Doank
wadidoank
user