Language model for wav2vec2.0 decoding
Hello, I implementedwav2vec2.0 codeand a language model is not used for decoding. How can I add a language model (let’s say a language model which is trained with KenLM) for decoding@patrickvonplaten?thanks in advance.Note: I also opened anissue, but redirected here.
2021-03-16T14:31:00Z
Hey Emre!Yeah good question - we currently don’t support evaluating with a language model, but we plan on adding this functionality soon! It’s sadly not that trivial to decode a CTC model with a language model. I’ll try to keep you posted for updates here!
2021-04-17T13:03:32Z
Assuming that one has a kenlm model already, am I wrong to assume that’s it’s just a matter of giving the wav2vec2 output logits as argument to the ctcdecode main function, exemplified here:GitHub - parlance/ctcdecode: PyTorch CTC Decoder bindings?Or is there more to it than that?
2021-04-19T07:56:27Z
@EmreOzkoseGood question i think. But i don’t start this as professional level. I’m currently searching on this.
2021-04-22T22:41:03Z
Hi all, I’ve been experimenting kenlm with wav2vec2 here is thenotebokI dont know if this is a proper implementation, but it works!I also still need to cleanup some stuff like vocab & other thing.
2021-04-23T07:39:49Z
@WikidepiaCan you share how much it improved your WER score? Also, did you tried character level LM as well?
2021-04-24T08:43:32Z
It improved from 14.2 to 9.2. I haven’t tried character level LM
2021-04-24T16:54:28Z
I added support for KenLM using the flashlight library here:Wav2Vec-Wrapper/test.py at main · Edresson/Wav2Vec-Wrapper · GitHubIt supports the use of the binary file instead of the arpa and it is also possible to restrict the model’s vocabulary.
2021-04-26T05:40:00Z
Thank you@Wikidepiaand@Edresson. I will check out.
2021-05-10T21:44:06Z
Hi Patrick!Any news on language model evaluation support?
2021-05-16T18:39:32Z
SeeAdded Feature: Prefix decoding for wav2vec2 models by deepang17 · Pull Request #11606 · huggingface/transformers · GitHub
2021-05-27T20:19:40Z
Wiki, when I apply your code, it predicts only spaces and -. Is there any reason for it?
2021-06-15T22:10:16Z
@jolurfyou can also use this decoder (GitHub - parlance/ctcdecode: PyTorch CTC Decoder bindings). Take the labels from your tokenizer and create a n-gram language model with KenLM. After that you can feed the logits from your Wav2Vec2 model into the decoder.@patrickvonplatenAre there any updates on the transformer language model?Voidful suggested to combine the wav2vec2 probabilities with those of the gpt2 model:(huggingface_notebook/xlsr_gpt.ipynb at main · voidful/huggingface_notebook · GitHub)However, the CTC Vocab seems to match the GPT Vocab. Unfortunately, this is not the case in English. Is there already a solution?
2021-06-17T14:09:46Z
If this discussion is still ongoing, then there is a pull requestAdded Feature: Prefix decoding for wav2vec2 models by deepang17 · Pull Request #11606 · huggingface/transformers · GitHubcurrently open, and as@ChristophBenschmentions a means of using KenLM fromGitHub - parlance/ctcdecode: PyTorch CTC Decoder bindings. We have an example of this atGitHub - techiaith/docker-wav2vec2-xlsr-ft-cy: Hyfforddi modelau adnabod lleferydd Cymraeg wav2vec2 a KenLM a'u darparu drwy weinydd gwasanaeth API // Train wav2vec2 and KenLM models for Welsh language speech recognition and/or provide via a simple API server.that’s reduced our WER score for Welsh from 25% to 15%. Since our scripts use HuggingFace’s OSCAR dataset, they should be easily adaptable to train and optimize LMs for other lesser resourced languages as well.
2021-06-24T08:11:07Z
Thanks@DewiBrynJonesfor the implementation, love the idea to have readmes in your local language and reference an English version
2021-07-01T08:09:13Z
Hi all! As advised by@andersgb1I used a kenlm n-gram language model on top of a distilled wav2vec2 that I trained and it improved my WER (26 → 12.6). If you guys are interested here’s the notebook (executes seamlessly on colab)OthmaneJ/distil-wav2vec2 · Hugging Face
2021-07-05T14:58:30Z
@OthmaneJCould you please share the code you used for distilling wav2vec2?
2021-07-05T15:23:36Z
So to use the wav2vec2 with gpt2 for English, would we have to just match the vocab used in the wav2vec2 with the vocab used in the gpt2?
2021-07-07T10:23:49Z
Is integrating an LM for wav2vec2 basically pointless now with the release of HuBERT? Which if I understand correctly is both an audio and language model at the same time?facebook/hubert-xlarge-ll60k · Hugging FaceI’m trying to achieve sub-5% (surpassing human performance) WER, but I don’t know if after I fine-tune this Hubert on my own data it will achieve that or not, because I’m not sure about the language model thing.Does it also need an integration with a language model to actually make it perform well?
2021-09-13T19:21:21Z
Can I create a character-level LM with KenLM?
No date available
No reply text available