How to create Wav2Vec2 With Language model
Now that language model boosted decoding is possible for Wav2Vec2 (https://twitter.com/PatrickPlaten/status/1468999507488788480andpatrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm · Hugging Faceit’s important to knowHow can one create a Wav2Vec2 + LM repo?.Let’s explain (hopefully this is simpler in the future):Install kenlm:The best guide to build kenlm is this github gist here:kenlm/BUILDING at master · kpu/kenlm · GitHubIMOCreate an ngram model:This is explained quite well here:GitHub - kpu/kenlm: KenLM: Faster and Smaller Language Model QueriesI wrote a short python script that allows to quickly create a ngram from a text corpus of common voice:#!/usr/bin/env python3
from datasets import load_dataset
import os
import argparse

parser = argparse.ArgumentParser()
parser.add_argument(
    "--language", default="polish", type=str, required=True, help="Language to run comparison on. Choose one of 'polish', 'portuguese', 'spanish' or add more to this script."
)
parser.add_argument(
    "--path_to_ngram", type=str, required=True, help="Path to kenLM ngram"
)
args = parser.parse_args()

ds = load_dataset("multilingual_librispeech", f"{args.language}", split="train")

with open("text.txt", "w") as f:
    f.write(" ".join(ds["text"]))

os.system(f"./kenlm/build/bin/lmplz -o 5 <text.txt > {args.path_to_ngram}")

## VERY IMPORTANT!!!:
# After the language model is created, one should open the file. one should add a `</s>`
# The file should have a structure which looks more or less as follows:

# \data\
# ngram 1=86586
# ngram 2=546387
# ngram 3=796581
# ngram 4=843999
# ngram 5=850874

# \1-grams:
# -5.7532206      <unk>   0
# 0       <s>     -0.06677356
# -3.4645514      drugi   -0.2088903
# ...

# Now it is very important also add a </s> token to the n-gram
# so that it can be correctly loaded. You can simple copy the line:

# 0       <s>     -0.06677356

# and change <s> to </s>. When doing this you should also inclease `ngram` by 1.
# The new ngram should look as follows:

# \data\
# ngram 1=86587
# ngram 2=546387
# ngram 3=796581
# ngram 4=843999
# ngram 5=850874

# \1-grams:
# -5.7532206      <unk>   0
# 0       <s>     -0.06677356
# 0       </s>     -0.06677356
# -3.4645514      drugi   -0.2088903
# ...

# Now the ngram can be correctly used with `pyctcdecode`See:Wav2Vec2_PyCTCDecode/create_ngram.py at main · patrickvonplaten/Wav2Vec2_PyCTCDecode · GitHubFeel free to copy those lines of code. Multi-Lingual librispeech is already a very clean text corpus. You might want to pre-process other text corpora to not include any punctuation etc…As an example this step created the Spanish 5-gram here:kensho/5gram-spanish-kenLM · Hugging FaceNow we should load the language model in a PyCTCBeamSearchDecoder as this is the format we need. Here one should be very careful to choose exactly the same vocabulary as the Wav2Vec2’s tokenizer vocab.At first we should pick a fine-tuned Wav2Vec2 model that we would like to add a language model to.Let’s choose:jonatasgrosman/wav2vec2-large-xlsr-53-spanish · Hugging FaceNow we instantiate aBeamSearchDecoderand save it to a folderwav2vec2_with_lm.E.g. you can run this code:from transformers import AutoTokenizer
from pyctcdecode import build_ctcdecoder

tokenizer = AutoTokenizer.from_pretrained("jonatasgrosman/wav2vec2-large-xlsr-53-spanish")

vocab_dict = tokenizer.get_vocab()
sorted_dict = {k: v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

decoder = build_ctcdecoder(
    list(sorted_dict.keys()),
    args.path_to_ngram,
)

decoder.save_to_dir("wav2vec2_with_lm")Now we should have saved the following files inwav2vec2_with_lm:- language_model
  - attrs.json
  - kenLM.arpa
  - unigrams.txt
- alphabet.jsonThat’s it! Now all you need to do is to upload this to your Wav2Vec2 model so that the directory structure looks as follows:huggingface.copatrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Now, all you need to change is your decoding to:huggingface.copatrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2021-12-10T16:07:39Z
This is cool!Can you make the inference widget use the language model?
2021-12-17T06:47:20Z
This is awesome!I have a bunch of KenLM models for different languages trained on sentence piece-tokenized Wikipedia and OSCAR here if you find them useful:edugp/kenlm at main
2021-12-17T18:32:40Z
Hi@patrickvonplatenWere you able to install kenlm in spaces?
2021-12-18T09:46:41Z
This is awesome. Thank you!
2021-12-20T14:59:17Z
@Harveenchadhayou can install it adding this line to your requirements.txt:https://github.com/kpu/kenlm/archive/master.zip
2021-12-22T13:51:01Z
boris:inferenceYes working on it!
2022-01-02T18:16:26Z
If someone is trying this with LM like me and has problems with AutoProcessor / AutoModelForCTCinstall transformers from@patrickvonplatencommit[AutoProcessor] Correct AutoProcessor and automatically add processor… · huggingface/transformers@a139288 · GitHubAt least I was able to load my model (RASMUS/wav2vec2-xlsr · Hugging Face) with LM after that fix
2022-01-07T11:30:33Z
Hi,Thank you for sharing resources !I have come across the paper of wav2vec2, and according to the paper it seems like using a Transformer as a language model yields the best results in terms of WER.However, I do not find any other reference/resource on the subject, nor did I find the weights of the Transformer used for decoding.Is it possible that you might have any resource on the subject? (where I could find the weights of such transformer decoder)Thanks a lot for helping the community
2022-01-27T16:58:48Z
patrickvonplaten:decoder.save_to_dir("wav2vec2_with_lm")While predicting the text with  Model with LM , i am getting below error.image1291×511 52.6 KB
2022-01-29T14:06:16Z
Hey@yaswanth,Sorry to reply so late. Could you add a fully reproducible code sample? This is as minimal as possible?
2022-01-29T14:58:41Z
code:image1372×547 67.4 KBModel_Files:Error:image1310×581 49.9 KB
2022-05-10T16:13:15Z
Hi@patrickvonplaten@philschmidIs it possible to deploy Wav2Vec2 with a KenLM Language Model in Amazon SageMaker? I was following this articlehttps://www.philschmid.de/automatic-speech-recognition-sagemakerbut i couldn’t find an option in the sagemaker HF library to use a language model.
2022-05-10T17:10:01Z
Hey@diegoseto,Yes, it is possible by deploying your model from Amazon S3 with a custominference.pyincluding the inference code.Here is anexamplenotebook for using asentence-transformers. You just need to modify theinference.pyand how themodel.tar.gzis created.github.comhuggingface/notebooks/blob/main/sagemaker/17_custom_inference_script/sagemaker-notebook.ipynb{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentence Embeddings with Hugging Face Transformers, Sentence Transformers and Amazon SageMaker - Custom Inference for creating document embeddings with Hugging Face's Transformers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a1644f1",
   "metadata": {},
   "source": [
    "Welcome to this getting started guide. We will use the Hugging Face Inference DLCs and Amazon SageMaker Python SDK to create a [real-time inference endpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html) running a Sentence Transformers for document embeddings. Currently, the [SageMaker Hugging Face Inference Toolkit](https://github.com/aws/sagemaker-huggingface-inference-toolkit) supports the [pipeline feature](https://huggingface.co/transformers/main_classes/pipelines.html) from Transformers for zero-code deployment. This means you can run compatible Hugging Face Transformer models without providing pre- & post-processing code. Therefore we only need to provide an environment variable `HF_TASK` and `HF_MODEL_ID` when creating our endpoint and the Inference Toolkit will take care of it. This is a great feature if you are working with existing [pipelines](https://huggingface.co/transformers/main_classes/pipelines.html).\n",
    "\n",
    "If you want to run other tasks, such as creating document embeddings, you can the pre- and post-processing code yourself, via an `inference.py` script. The Hugging Face Inference Toolkit allows the user to override the default methods of the `HuggingFaceHandlerService`.\n",
    "\n",
    "The custom module can override the following methods:\n",
    "\n",This file has been truncated.show original
2022-05-10T17:45:34Z
Cool. Thank you, i will try this. Thanks for the quick reply too
2023-05-05T18:17:34Z
Hi everyone - I’m pretty sure I know the answer to this but just making sure. Has anyone managed to use the processor with a kenlm language model on Windows?  I created my .arpa file on a Linux os hoping to be able to use it with wav2vec2 on Windows without actually building/installing kenlm, but this does not appear to be the case. Even when I create the processor, bundled with the lm, on Linux and then load the processor from_pretrained on my Windows script (based on the blog), it still wants to call kenlm and crashes. Just want to confirm that there is no way around this and that, at this time, Linux is mandatory for using wav2vec2 with a language model, even with an already existing .arpa file that has been created somewhere else.thanks!Jonathan