Wav2Vec2 WER remains 1.00 and return blank transcriptions
Hello everyone. I faced a strange problem and not sure about how i can resolve the problem myself.I was fine tuning wav2vec2 on two custom dataset and although previously it used to work fine now I am not sure what changed.Issue#1 : The WER does not decrease, it always remains constant at 1.00 WER .Issue#2 : When you make predictions you get blank transciptions.It happened twice and then i decided to run exactly on English Fine tuning blog post.Fine-Tune Wav2Vec2 for English ASR withTransformersI am attaching an image below, the only change i made is on train and eval steps to experiment faster.Please run the following colab notebook to reproduce results :Google Colab
2021-12-12T14:26:58Z
Hey thereI was experiencing the same problem on my own dataset. I managed to solve it by using librosa to load the raw data.librosa.load(filename, sr = 16000)Previously i was using pydub.AudioSegment
2022-01-22T18:32:32Z
Same issue. Were you able to figure it out?
2022-01-25T03:28:12Z
Can you share a detailed configuration. Also share the colab notebook if possible ??
2022-01-27T18:10:05Z
absl-py==0.12.0aiohttp==3.8.0aiosignal==1.2.0alabaster @ file:///home/ktietz/src/ci/alabaster_1611921544520/workanaconda-client==1.7.2anaconda-navigator==2.0.3anaconda-project @ file:///tmp/build/80754af9/anaconda-project_1610472525955/workanyio @ file:///tmp/build/80754af9/anyio_1617783275907/work/distappdirs==1.4.4argh==0.26.2argon2-cffi @ file:///tmp/build/80754af9/argon2-cffi_1613037097816/workasn1crypto @ file:///tmp/build/80754af9/asn1crypto_1596577642040/workastroid @ file:///tmp/build/80754af9/astroid_1613500854201/workastropy @ file:///tmp/build/80754af9/astropy_1617745353437/workastunparse==1.6.3async-generator @ file:///home/ktietz/src/ci/async_generator_1611927993394/workasync-timeout==4.0.0atomicwrites==1.4.0attrs @ file:///tmp/build/80754af9/attrs_1604765588209/workaudioread==2.1.9autopep8 @ file:///tmp/build/80754af9/autopep8_1615918855173/workBabel @ file:///tmp/build/80754af9/babel_1607110387436/workbackcall @ file:///home/ktietz/src/ci/backcall_1611930011877/workbackports.functools-lru-cache @ file:///tmp/build/80754af9/backports.functools_lru_cache_1618170165463/workbackports.shutil-get-terminal-size @ file:///tmp/build/80754af9/backports.shutil_get_terminal_size_1608222128777/workbackports.tempfile @ file:///home/linux1/recipes/ci/backports.tempfile_1610991236607/workbackports.weakref==1.0.post1beautifulsoup4 @ file:///home/linux1/recipes/ci/beautifulsoup4_1610988766420/workbitarray @ file:///tmp/build/80754af9/bitarray_1620827551536/workbkcharts==0.2black==19.10b0bleach @ file:///tmp/build/80754af9/bleach_1612211392645/workbokeh @ file:///tmp/build/80754af9/bokeh_1620779595936/workboto==2.49.0Bottleneck==1.3.2brotlipy==0.7.0cachetools==4.2.4certifi==2020.12.5cffi @ file:///tmp/build/80754af9/cffi_1613246945912/workchardet @ file:///tmp/build/80754af9/chardet_1607706746162/workcharset-normalizer==2.0.7clang==5.0click @ file:///home/linux1/recipes/ci/click_1610990599742/workcloudpickle @ file:///tmp/build/80754af9/cloudpickle_1598884132938/workclyent==1.2.2colorama @ file:///tmp/build/80754af9/colorama_1607707115595/workconda==4.10.1conda-build==3.21.4conda-content-trust @ file:///tmp/build/80754af9/conda-content-trust_1617045594566/workconda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1618262148928/workconda-repo-cli @ file:///tmp/build/80754af9/conda-repo-cli_1620168426516/workconda-token @ file:///tmp/build/80754af9/conda-token_1620076980546/workcontextlib2==0.6.0.post1cryptography @ file:///tmp/build/80754af9/cryptography_1616769286105/workcycler==0.10.0Cython @ file:///tmp/build/80754af9/cython_1618435160151/workcytoolz==0.11.0dask @ file:///tmp/build/80754af9/dask-core_1617390489108/workdatasets==1.4.1decorator @ file:///tmp/build/80754af9/decorator_1617916966915/workdefusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/workdiff-match-patch @ file:///tmp/build/80754af9/diff-match-patch_1594828741838/workdill==0.3.4distributed @ file:///tmp/build/80754af9/distributed_1620902833129/workdocutils @ file:///tmp/build/80754af9/docutils_1620827984873/workentrypoints==0.3et-xmlfile==1.0.1fastcache==1.1.0ffmpeg==1.4filelock @ file:///home/linux1/recipes/ci/filelock_1610993975404/workflake8 @ file:///tmp/build/80754af9/flake8_1615834841867/workFlask @ file:///home/ktietz/src/ci/flask_1611932660458/workflatbuffers==1.12frozendict==2.1.1frozenlist==1.2.0fsspec==2021.10.1future==0.18.2gast==0.4.0gdown==4.2.0gevent @ file:///tmp/build/80754af9/gevent_1616770671827/workglob2 @ file:///home/linux1/recipes/ci/glob2_1610991677669/workgmpy2==2.0.8google-auth==2.3.3google-auth-oauthlib==0.4.6google-pasta==0.2.0googleapis-common-protos==1.53.0greenlet @ file:///tmp/build/80754af9/greenlet_1611957705398/workgrpcio==1.41.1h5py==3.1.0HeapDict==1.0.1html5lib @ file:///tmp/build/80754af9/html5lib_1593446221756/workhuggingface-hub==0.0.2idna @ file:///home/linux1/recipes/ci/idna_1610986105248/workimageio @ file:///tmp/build/80754af9/imageio_1617700267927/workimagesize @ file:///home/ktietz/src/ci/imagesize_1611921604382/workimportlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1617874469820/workiniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/workintervaltree @ file:///tmp/build/80754af9/intervaltree_1598376443606/workipykernel @ file:///tmp/build/80754af9/ipykernel_1596207638929/work/dist/ipykernel-5.3.4-py3-none-any.whlipython @ file:///tmp/build/80754af9/ipython_1617120885885/workipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/workipywidgets @ file:///tmp/build/80754af9/ipywidgets_1610481889018/workisort @ file:///tmp/build/80754af9/isort_1616355431277/workitsdangerous @ file:///home/ktietz/src/ci/itsdangerous_1611932585308/workjdcal==1.4.1jedi @ file:///tmp/build/80754af9/jedi_1606932564285/workjeepney @ file:///tmp/build/80754af9/jeepney_1606148855031/workJinja2 @ file:///tmp/build/80754af9/jinja2_1612213139570/workjiwer==2.2.1joblib @ file:///tmp/build/80754af9/joblib_1613502643832/workjson5==0.9.5jsonschema @ file:///tmp/build/80754af9/jsonschema_1602607155483/workjupyter==1.0.0jupyter-client @ file:///tmp/build/80754af9/jupyter_client_1616770841739/workjupyter-console @ file:///tmp/build/80754af9/jupyter_console_1616615302928/workjupyter-core @ file:///tmp/build/80754af9/jupyter_core_1612213311222/workjupyter-packaging @ file:///tmp/build/80754af9/jupyter-packaging_1613502826984/workjupyter-server @ file:///tmp/build/80754af9/jupyter_server_1616083640759/workjupyterlab @ file:///tmp/build/80754af9/jupyterlab_1619133235951/workjupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/workjupyterlab-server @ file:///tmp/build/80754af9/jupyterlab_server_1617134334258/workjupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/workkeras==2.6.0Keras-Preprocessing==1.1.2keyring @ file:///tmp/build/80754af9/keyring_1614616740399/workkiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282420641/worklazy-object-proxy @ file:///tmp/build/80754af9/lazy-object-proxy_1616526917483/worklibarchive-c @ file:///tmp/build/80754af9/python-libarchive-c_1617780486945/worklibrosa==0.8.1llvmlite==0.36.0locket==0.2.1lxml @ file:///tmp/build/80754af9/lxml_1616443220220/workMarkdown==3.3.4MarkupSafe==1.1.1matplotlib @ file:///tmp/build/80754af9/matplotlib-suite_1613407855456/workmccabe==0.6.1mistune==0.8.4mkl-fft==1.3.0mkl-random @ file:///tmp/build/80754af9/mkl_random_1618853849286/workmkl-service==2.3.0mock @ file:///tmp/build/80754af9/mock_1607622725907/workmore-itertools @ file:///tmp/build/80754af9/more-itertools_1613676688952/workmpmath==1.2.1msgpack @ file:///tmp/build/80754af9/msgpack-python_1612287151062/workmultidict==5.2.0multipledispatch==0.6.0multiprocess==0.70.12.2mypy-extensions==0.4.3navigator-updater==0.2.1nbclassic @ file:///tmp/build/80754af9/nbclassic_1616085367084/worknbclient @ file:///tmp/build/80754af9/nbclient_1614364831625/worknbconvert @ file:///tmp/build/80754af9/nbconvert_1601914830498/worknbformat @ file:///tmp/build/80754af9/nbformat_1617383369282/worknemo-toolkit==1.5.0nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/worknetworkx @ file:///tmp/build/80754af9/networkx_1598376031484/worknltk @ file:///tmp/build/80754af9/nltk_1618327084230/worknose @ file:///tmp/build/80754af9/nose_1606773131901/worknotebook @ file:///tmp/build/80754af9/notebook_1616443462982/worknumba @ file:///tmp/build/80754af9/numba_1616774046117/worknumexpr @ file:///tmp/build/80754af9/numexpr_1618856167419/worknumpy==1.19.5numpydoc @ file:///tmp/build/80754af9/numpydoc_1605117425582/workoauthlib==3.1.1olefile==0.46onnx==1.10.2opencv-python==4.5.4.58openpyxl @ file:///tmp/build/80754af9/openpyxl_1615411699337/workopt-einsum==3.3.0packaging @ file:///tmp/build/80754af9/packaging_1611952188834/workpandas==1.2.4pandocfilters @ file:///tmp/build/80754af9/pandocfilters_1605120460739/workparso==0.7.0partd @ file:///tmp/build/80754af9/partd_1618000087440/workpath @ file:///tmp/build/80754af9/path_1614022220526/workpathlib2 @ file:///tmp/build/80754af9/pathlib2_1607024983162/workpathspec==0.7.0patsy==0.5.1pep8==1.7.1pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/workpickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/workPillow @ file:///tmp/build/80754af9/pillow_1617383569452/workpkginfo==1.7.0pluggy @ file:///tmp/build/80754af9/pluggy_1615976321666/workply==3.11pooch==1.5.2prometheus-client @ file:///tmp/build/80754af9/prometheus_client_1618088486455/workpromise==2.3prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1616415428029/workprotobuf==3.19.1psutil @ file:///tmp/build/80754af9/psutil_1612298023621/workptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whlpy @ file:///tmp/build/80754af9/py_1607971587848/workpyarrow==6.0.0pyasn1==0.4.8pyasn1-modules==0.2.8pycodestyle @ file:///home/ktietz/src/ci_mi/pycodestyle_1612807597675/workpycosat==0.6.3pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/workpycurl==7.43.0.6pyDeprecate==0.3.1pydocstyle @ file:///tmp/build/80754af9/pydocstyle_1616182067796/workpydub==0.25.1pyerfa @ file:///tmp/build/80754af9/pyerfa_1619390903914/workpyflakes @ file:///home/ktietz/src/ci_ipy2/pyflakes_1612551159640/workPygments @ file:///tmp/build/80754af9/pygments_1615143339740/workpylint @ file:///tmp/build/80754af9/pylint_1617135829881/workpyls-black @ file:///tmp/build/80754af9/pyls-black_1607553132291/workpyls-spyder @ file:///tmp/build/80754af9/pyls-spyder_1613849700860/workpyodbc===4.0.0-unsupportedpyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/workpyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/workpyrsistent @ file:///tmp/build/80754af9/pyrsistent_1600141720057/workPySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/workpytest==6.2.3python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/workpython-jsonrpc-server @ file:///tmp/build/80754af9/python-jsonrpc-server_1600278539111/workpython-language-server @ file:///tmp/build/80754af9/python-language-server_1607972495879/workpython-Levenshtein==0.12.2pytorch-lightning==1.5.3pytz @ file:///tmp/build/80754af9/pytz_1612215392582/workPyWavelets @ file:///tmp/build/80754af9/pywavelets_1601658317819/workpyxdg @ file:///tmp/build/80754af9/pyxdg_1603822279816/workPyYAML==5.4.1pyzmq==20.0.0QDarkStyle==2.8.1QtAwesome @ file:///tmp/build/80754af9/qtawesome_1615991616277/workqtconsole @ file:///tmp/build/80754af9/qtconsole_1616775094278/workQtPy==1.9.0regex @ file:///tmp/build/80754af9/regex_1617569202463/workrequests @ file:///tmp/build/80754af9/requests_1608241421344/workrequests-oauthlib==1.3.0resampy==0.2.2rope @ file:///tmp/build/80754af9/rope_1602264064449/workrsa==4.7.2Rtree @ file:///tmp/build/80754af9/rtree_1618420845272/workruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016699510/workruamel.yaml==0.17.17ruamel.yaml.clib==0.2.6sacremoses==0.0.46scikit-image==0.18.1scikit-learn @ file:///tmp/build/80754af9/scikit-learn_1614446682169/workscipy @ file:///tmp/build/80754af9/scipy_1618855647378/workseaborn @ file:///tmp/build/80754af9/seaborn_1608578541026/workSecretStorage @ file:///tmp/build/80754af9/secretstorage_1614022784285/workSend2Trash @ file:///tmp/build/80754af9/send2trash_1607525499227/worksentencepiece==0.1.96simplegeneric==0.8.1singledispatch @ file:///tmp/build/80754af9/singledispatch_1614366001199/worksip==4.19.13six @ file:///tmp/build/80754af9/six_1605205327372/worksniffio @ file:///tmp/build/80754af9/sniffio_1614030475067/worksnowballstemmer @ file:///tmp/build/80754af9/snowballstemmer_1611258885636/worksortedcollections @ file:///tmp/build/80754af9/sortedcollections_1611172717284/worksortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1606865132123/workSoundFile==0.10.3.post1soupsieve @ file:///tmp/build/80754af9/soupsieve_1616183228191/workSphinx @ file:///tmp/build/80754af9/sphinx_1620777493457/worksphinxcontrib-applehelp @ file:///home/ktietz/src/ci/sphinxcontrib-applehelp_1611920841464/worksphinxcontrib-devhelp @ file:///home/ktietz/src/ci/sphinxcontrib-devhelp_1611920923094/worksphinxcontrib-htmlhelp @ file:///home/ktietz/src/ci/sphinxcontrib-htmlhelp_1611920974801/worksphinxcontrib-jsmath @ file:///home/ktietz/src/ci/sphinxcontrib-jsmath_1611920942228/worksphinxcontrib-qthelp @ file:///home/ktietz/src/ci/sphinxcontrib-qthelp_1611921055322/worksphinxcontrib-serializinghtml @ file:///home/ktietz/src/ci/sphinxcontrib-serializinghtml_1611920755253/worksphinxcontrib-websupport @ file:///tmp/build/80754af9/sphinxcontrib-websupport_1597081412696/workspyder @ file:///tmp/build/80754af9/spyder_1616775618138/workspyder-kernels @ file:///tmp/build/80754af9/spyder-kernels_1614030590686/workSQLAlchemy @ file:///tmp/build/80754af9/sqlalchemy_1620712430742/workstatsmodels @ file:///tmp/build/80754af9/statsmodels_1614023746358/worksympy @ file:///tmp/build/80754af9/sympy_1618252284338/worktables==3.6.1tblib @ file:///tmp/build/80754af9/tblib_1597928476713/worktensorboard==2.7.0tensorboard-data-server==0.6.1tensorboard-plugin-wit==1.8.0tensorflow==2.6.1tensorflow-addons==0.14.0tensorflow-datasets==3.2.1tensorflow-estimator==2.6.0tensorflow-metadata==1.4.0termcolor==1.1.0terminado==0.9.4testpath @ file:///home/ktietz/src/ci/testpath_1611930608132/worktextdistance @ file:///tmp/build/80754af9/textdistance_1612461398012/worktf2crf==0.1.33threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whlthree-merge @ file:///tmp/build/80754af9/three-merge_1607553261110/worktifffile==2020.10.1tokenizers==0.10.3toml @ file:///tmp/build/80754af9/toml_1616166611790/worktoolz @ file:///home/linux1/recipes/ci/toolz_1610987900194/worktorch @ file:///home/omer/torch-1.8.0.dev20210208%2Bcu110-cp38-cp38-linux_x86_64.whltorchaudio @ file:///home/omer/torchaudio-0.8.0.dev20210208-cp38-cp38-linux_x86_64.whltorchmetrics==0.6.0torchtext @ file:///home/omer/torchtext-0.9.0.dev20210208-cp38-cp38-linux_x86_64.whltorchvision @ file:///home/omer/torchvision-0.9.0.dev20210208%2Bcu110-cp38-cp38-linux_x86_64.whltornado @ file:///tmp/build/80754af9/tornado_1606942300299/worktqdm==4.49.0traitlets @ file:///home/ktietz/src/ci/traitlets_1611929699868/worktransformers==4.4.0typed-ast @ file:///tmp/build/80754af9/typed-ast_1610484547928/worktypeguard==2.13.0typing-extensions @ file:///home/ktietz/src/ci_mi/typing_extensions_1612808209620/workujson @ file:///tmp/build/80754af9/ujson_1611259522456/workunicodecsv==0.14.1Unidecode==1.3.2urduhack==1.1.1urllib3 @ file:///tmp/build/80754af9/urllib3_1615837158687/workwatchdog @ file:///tmp/build/80754af9/watchdog_1612471027849/workwcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/workwebencodings==0.5.1Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/workwget==3.2widgetsnbextension==3.5.1wrapt==1.12.1wurlitzer @ file:///tmp/build/80754af9/wurlitzer_1617224664226/workxlrd @ file:///tmp/build/80754af9/xlrd_1608072521494/workXlsxWriter @ file:///tmp/build/80754af9/xlsxwriter_1617224712951/workxlwt==1.3.0xmltodict==0.12.0xxhash==2.0.2yapf @ file:///tmp/build/80754af9/yapf_1615749224965/workyarl==1.7.2zict==2.0.0zipp @ file:///tmp/build/80754af9/zipp_1615904174917/workzope.event==4.5.0zope.interface @ file:///tmp/build/80754af9/zope.interface_1616357211867/workColab Notebook
2022-01-27T18:11:58Z
I’ve also tried this on different batch sizes and more epochs. Getting the same result. Oddly though, the same notebook used to work a while back.
2022-10-20T00:36:40Z
I found that I had this problem when I had too many trainable parameters and not enough training data. It might help to freeze more parameters in the model.
2022-11-23T02:57:47Z
Did you fix this? I also found this problem, the same code but the different results…
2023-01-10T01:17:56Z
I am also facing the same issue. I am fine-tuning thefacebook/wav2vec2-xls-r-300mmodel over a low-resource language dataset. The WER metric does not change from 1.00 and the predictions are empty strings. I am using transformers==4.24.0 . I have tried setting the parameterdo_normalizationofWav2Vec2FeatureExtractorto False, as there was an associated issue ongithub. However, this didn’t work out.This is my model:model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-xls-r-300m",
    ctc_loss_reduction="mean",
    bos_token_id=processor.tokenizer.bos_token_id,
    eos_token_id=processor.tokenizer.eos_token_id,
    pad_token_id=processor.tokenizer.pad_token_id,
    vocab_size=len(processor.tokenizer),
)
model.freeze_feature_encoder()These are my training args:training_args = TrainingArguments(
  output_dir=repo_name,
  do_train = True,
  learning_rate=9.24e-05,
  # params for evaluation
  do_eval=True,
  evaluation_strategy = "epoch",
  group_by_length=True,
  per_device_train_batch_size=16,
  gradient_accumulation_steps=2,
  num_train_epochs=30,
  gradient_checkpointing=True,
  #fp16=True, # available for cuda
  save_steps=400,
  warmup_steps=500,
  save_total_limit=2,
  push_to_hub=True,
  overwrite_output_dir = True,
  # parameter for logs per epochs
  logging_strategy="epoch",
  logging_dir='./logs', 
  log_level = "debug",
  report_to="wandb", 
  run_name="quechua-30",
  # params for logs per steps
  #eval_steps=400,
  #logging_steps=400,
)trainer = Trainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=quechua["train"],
    eval_dataset=quechua["dev"],
    tokenizer=processor.feature_extractor,
    callbacks=[MyCallback]
)
trainer.train()
2023-02-09T00:33:45Z
Hello!May I ask did you solve the problem?I have the exact same issue that you wrote. Had search a lot, but didn’t find solutions that worked out.
2023-02-13T17:54:52Z
Hi all. I experienced this problem too.  In my case I’m pretty sure this is what happens when you are badly over-training the model. If you look at your logits in the output, you’ll probably find that the first logit is always largest, which I think is code for “nothing”.  I tried training using a very very small dataset (maybe 100 cases), logging at each step.  For the first couple of steps, the WER is below 1, then it eventually goes to 1 and gives that empty transcript, suggesting over-training. So, at the moment I’m doing a hyperparameter search to identify a stable learning rate and weight decay.  In a nutshell, I think the answer is probably to turn down your learning rate.
2023-10-20T09:00:04Z
Same issue getting 1 WER
2024-01-30T06:57:16Z
Any updates?I am also getting the same issue.WER remains at 1 during training, and I’m getting empty transcripts during inference.