espnet audio self-supervised-learning speech-recognition afr amh ara asm ast azj bel ben bos cat ceb cmn ces cym dan deu ell eng spa est fas ful fin tgl fra gle glg guj hau heb hin hrv hun hye ind ibo isl ita jpn jav kat kam kea kaz khm kan kor ckb kir ltz lug lin lao lit luo lav mri mkd mal mon mar msa mlt mya nob npi nld nso nya oci orm ory pan pol pus por ron rus bul snd slk slv sna som srp swe swh tam tel tgk tha tur ukr umb urd uzb vie wol xho yor yue zul dataset:fleurs dataset:babel dataset:voxpopuli dataset:commonvoice arxiv:2309.15317 arxiv:2305.10615 arxiv:1804.00015 license:cc-by-4.0 region:us
---
tags:
- espnet
- audio
- self-supervised-learning
- speech-recognition
multilinguality:
  - multilingual
task_categories:
  - automatic-speech-recognition
language:
  - afr
  - amh
  - ara
  - asm
  - ast
  - azj
  - bel
  - ben
  - bos
  - cat
  - ceb
  - cmn
  - ces
  - cym
  - dan
  - deu
  - ell
  - eng
  - spa
  - est
  - fas
  - ful
  - fin
  - tgl
  - fra
  - gle
  - glg
  - guj
  - hau
  - heb
  - hin
  - hrv
  - hun
  - hye
  - ind
  - ibo
  - isl
  - ita
  - jpn
  - jav
  - kat
  - kam
  - kea
  - kaz
  - khm
  - kan
  - kor
  - ckb
  - kir
  - ltz
  - lug
  - lin
  - lao
  - lit
  - luo
  - lav
  - mri
  - mkd
  - mal
  - mon
  - mar
  - msa
  - mlt
  - mya
  - nob
  - npi
  - nld
  - nso
  - nya
  - oci
  - orm
  - ory
  - pan
  - pol
  - pus
  - por
  - ron
  - rus
  - bul
  - snd
  - slk
  - slv
  - sna
  - som
  - srp
  - swe
  - swh
  - tam
  - tel
  - tgk
  - tha
  - tur
  - ukr
  - umb
  - urd
  - uzb
  - vie
  - wol
  - xho
  - yor
  - yue
  - zul
datasets:
- fleurs
- babel
- voxpopuli
- commonvoice
license: cc-by-4.0
---

## WavLabLM-MS 40k

[Paper](https://arxiv.org/abs/2309.15317)

This model was trained by [William Chen](https://wanchichen.github.io/) using ESPNet2's SSL recipe in [espnet](https://github.com/espnet/espnet/).
WavLabLM is an self-supervised audio encoder pre-trained on 40,000 hours of multilingual data across 136 languages. This specific variant, WavLabLM-MS, went through a second stage of pre-training on a balanced subset of the data to improve performance on lower-resource languages.
It achieves comparable performance to XLS-R 128 on the [ML-SUPERB Benchmark](https://arxiv.org/abs/2305.10615) with only 10% of the pre-training data.

```BibTex
@misc{chen2023joint,
      title={Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning}, 
      author={William Chen and Jiatong Shi and Brian Yan and Dan Berrebbi and Wangyou Zhang and Yifan Peng and Xuankai Chang and Soumi Maiti and Shinji Watanabe},
      year={2023},
      eprint={2309.15317},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


### Citing ESPnet

```BibTex
@inproceedings{watanabe2018espnet,
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
  title={{ESPnet}: End-to-End Speech Processing Toolkit},
  year={2018},
  booktitle={Proceedings of Interspeech},
  pages={2207--2211},
  doi={10.21437/Interspeech.2018-1456},
  url={http://dx.doi.org/10.21437/Interspeech.2018-1456}
}


```

or arXiv:

```bibtex
@misc{watanabe2018espnet,
  title={ESPnet: End-to-End Speech Processing Toolkit}, 
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
  year={2018},
  eprint={1804.00015},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}
```
