SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training
Paper
ā¢
2412.15649
ā¢
Published
SpeechColab
non-profit
AI & ML interests
Machine Learning for Audio/Speech
Recent Activity
View all activity
speechcolab's activity
yfyeungĀ
authored
2
papers
2 days ago
yfyeungĀ
authored
a
paper
about 1 month ago
yfyeungĀ
authored
9
papers
4 months ago
LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization
Paper
ā¢
2409.00819
ā¢
Published
Zipformer: A faster and better encoder for automatic speech recognition
Paper
ā¢
2310.11230
ā¢
Published
VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech
Paper
ā¢
2401.14321
ā¢
Published
An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
Paper
ā¢
2402.08846
ā¢
Published
ā¢
1
PromptASR for contextualized ASR with controllable style
Paper
ā¢
2309.07414
ā¢
Published
Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context
Paper
ā¢
2309.08105
ā¢
Published
GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement
Paper
ā¢
2406.11546
ā¢
Published
Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS
Paper
ā¢
2309.07377
ā¢
Published
Blank-regularized CTC for Frame Skipping in Neural Transducer
Paper
ā¢
2305.11558
ā¢
Published
jimbozhangĀ
authored
2
papers
7 months ago
jimbozhangĀ
authored
a
paper
10 months ago
sanchit-gandhiĀ
posted
an
update
10 months ago
Post
Why does returning timestamps help Whisper reduce hallucinations? š§
Empirically, most practitioners have found that setting
But why does this work?..
My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
Where we have a repeated hallucination for āon theā. If we ask the model to predict timestamps, then the āon theā has to contribute to the overall segment-level timing, e.g.:
However, itās impossible to fit 3 copies of āon theā within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper Robust Speech Recognition via Large-Scale Weak Supervision (2212.04356) ā it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).
Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?
Empirically, most practitioners have found that setting
return_timestamps=True helps reduce hallucinations, particularly when doing long-form evaluation with Transformersā āchunkedā algorithm. But why does this work?..
My interpretation is that forcing the model to predict timestamps is contradictory to hallucinations. Suppose you have the transcription:
The cat sat on the on the on the mat.Where we have a repeated hallucination for āon theā. If we ask the model to predict timestamps, then the āon theā has to contribute to the overall segment-level timing, e.g.:
<|0.00|> The cat sat on the on the on the mat.<|5.02|>However, itās impossible to fit 3 copies of āon theā within the time allocation given to the segment, so the probability for this hallucinatory sequence becomes lower, and the model actually predicts the correct transcription with highest probability:
<|0.00|> The cat sat on the mat.<|5.02|>In this sense, the end timestamp is of the opposite of the initial timestamp constraint they describe in Section 4.5 of the paper Robust Speech Recognition via Large-Scale Weak Supervision (2212.04356) ā it helps the model remove extra words at the end of the sequence (rather than the initial timestamp which helps when the model ignores words at the start), but the overall principle is the same (using timestamps to improve the probability of more realistic sequences).
Leaving it open to you: why do you think timestamps reduces Whisper hallucinations?
-
- 11 replies
sanchit-gandhiĀ
authored
a
paper
about 1 year ago
sanchit-gandhiĀ
authored
2
papers
over 1 year ago
sanchit-gandhiĀ
authored
a
paper
almost 2 years ago