Wav2vec2-base task performance
@patrickvonplatenThank you for the great work on releasing many variants of wav2vec2 and tutorials. They are super helpful. I am new to the ASR domain and able to reproduce some of the results with the released models. I was trying to compare the WER to the paper. I noticed there is a gap between HF’s models and the paper scores. I couldn’t figure out where the gap was coming from. Putting them in the table below, could you shed some light? Thank you!ModelPretraining DatasetFine Tuning DatasetEval. DatasetWER (%)Relative (%)Wav2vec 2.0 - Table 1, 3rd row   from bottomLS-960Labelled LS-100clean/test2.60baselinefacebook/wav2vec2-base-100hLS-960Labelled LS-100clean/test6.10235%Wav2vec 2.0 - Table   1, 3rd row   from bottomLS-960Labelled LS-100other/test6.3baselinefacebook/wav2vec2-base-100hLS-960Labelled   LS-100other/test13.5214%Wav2vec 2.0 - Table 2, 3rd   row   from bottomLS-960Labelled   LS-960clean/test2.1baselinefacebook/wav2vec2-base-960hLS-960Labelled   LS-960clean/test3.4162%Wav2vec 2.0 - Table 2, 3rd   row   from bottomLS-960Labelled   LS-960other/test4.8baselinefacebook/wav2vec2-base-960hLS-960Labelled   LS-960other/test8.6179%
2022-02-21T16:29:35Z
I think maybe that’s because in the paper they add a transformer LM to improve the performance. By the way, I am wondering if there is any discussion about how to combine a transformer LM with huggingface’s wav2vev2 model as well. I’ve found oneblog articlewritten by@patrickvonplatenshowing how to boost wav2vec2 with n-gram LM, but currently I don’t know how to combine the model with a transformer LM.
2022-03-18T16:29:19Z
It should actually be very easy to add a LM to Wav2Vec2, I have it done here:patrickvonplaten/wav2vec2-base-100h-with-lm · Hugging FaceAll you need to do is to take an official ngram, e.g. this one:openslr.organd then just follow the blog post here:Boosting Wav2Vec2 with n-grams in 🤗 Transformers
2022-03-18T16:29:38Z
The results without LM should match more or less - I’ve tested this for a couple of checkpoints
2023-05-08T18:46:01Z
Thanks@patrickvonplaten,@Kuray107for your comments. It appeared that I misunderstood and make incorrect correspondence. Updated table as follow. HF’s models are close to paperModelPretraining DatasetFine Tuning DatasetEval. DatasetWER (%)Relative (%)Wav2vec 2.0 - Table 9, 9th row from bottomLS-960Labelled LS-100test/clean6.1baselinefacebook/wav2vec2-base-100hLS-960Labelled LS-100test/clean6.10%Wav2vec 2.0 - Table 9, 9rd row from bottomLS-960Labelled LS-100test/other13.3baselinefacebook/wav2vec2-base-100hLS-960Labelled LS-100test/other13.52%Wav2vec 2.0 - Table 10, 9th row from bottomLS-960Labelled LS-960test/clean3.4baselinefacebook/wav2vec2-base-960hLS-960Labelled LS-960test/clean3.40%Wav2vec 2.0 - Table 10, 9th row from bottomLS-960Labelled LS-960test/other8.5baselinefacebook/wav2vec2-base-960hLS-960Labelled LS-960test/other8.61%