# Golos dataset

Golos is a Russian corpus suitable for speech research. The dataset mainly consists of recorded audio files manually annotated on the crowd-sourcing platform. The total duration of the audio is about 1240 hours. 
We have made the corpus freely available for downloading, along with the acoustic model prepared on this corpus. 
Also we create 3-gram KenLM language model using an open Common Crawl corpus.

## **Dataset structure**

| Domain         | Train files | Train hours  | Test files | Test hours |
|:--------------:|:----------:|:------:|:-----:|:----:|
| Crowd          | 979 796    | 1 095  | 9 994 | 11.2 |
| Farfield       | 124 003    |   132.4| 1 916 |  1.4 |
| Total          | 1 103 799  | 1 227.4|11 910 | 12.6 |


## **Downloads**


### **Audio files in opus format**

| Archive          | Size       |  Link               |
|:-----------------|:-----------|:--------------------|
| golos_opus.tar   | 20.5 GB    | https://sc.link/JpD |

### **Audio files in wav format**

Manifest files with all the training transcription texts are in the train_crowd9.tar archive listed in the table:

| Archives          | Size       |  Links              |
|-------------------|------------|---------------------|
| train_farfield.tar| 15.4 GB    | https://sc.link/1Z3 |
| train_crowd0.tar  | 11 GB      | https://sc.link/Lrg |
| train_crowd1.tar  | 14 GB      | https://sc.link/MvQ |
| train_crowd2.tar  | 13.2 GB    | https://sc.link/NwL |
| train_crowd3.tar  | 11.6 GB    | https://sc.link/Oxg |
| train_crowd4.tar  | 15.8 GB    | https://sc.link/Pyz |
| train_crowd5.tar  | 13.1 GB    | https://sc.link/Qz7 |
| train_crowd6.tar  | 15.7 GB    | https://sc.link/RAL |
| train_crowd7.tar  | 12.7 GB    | https://sc.link/VG5 |
| train_crowd8.tar  | 12.2 GB    | https://sc.link/WJW |
| train_crowd9.tar  | 8.08 GB    | https://sc.link/XKk |
| test.tar          | 1.3 GB     | https://sc.link/Kqr |


### **Acoustic and language models**

Acoustic model built using [QuartzNet15x5](https://arxiv.org/pdf/1910.10261.pdf) architecture and trained using [NeMo toolkit](https://github.com/NVIDIA/NeMo/tree/r1.0.0b4)


Three n-gram language models created using [KenLM Language Model Toolkit](https://kheafield.com/code/kenlm)

* LM built on [Common Crawl](https://commoncrawl.org) Russian dataset
* LM built on Golos train set
* LM built on [Common Crawl](https://commoncrawl.org) and Golos datasets together (50/50)

| Archives                 | Size       |  Links          |
|--------------------------|------------|-----------------|
| QuartzNet15x5_golos.nemo | 68 MB      | https://sc.link/ZMv |
| KenLMs.tar               | 4.8 GB     | https://sc.link/YL0  |


Golos data and models are also available in the hub of pre-trained models, datasets, and containers - DataHub ML Space. You can train the model and deploy it on the high-performance SberCloud infrastructure in [ML Space](https://sbercloud.ru/ru/aicloud/mlspace) - full-cycle machine learning development platform for DS-teams collaboration based on the Christofari Supercomputer.


## **Evaluation**

Percents of Word Error Rate for different test sets


| Decoder \ Test set    | Crowd test  | Farfield test    | MCV<sup>1</sup> dev | MCV<sup>1</sup> test |
|-------------------------------------|-----------|----------|-----------|----------|
| Greedy decoder                      | 4.389 %   | 14.949 % | 9.314 %   | 11.278 % |
| Beam Search with Common Crawl LM    | 4.709 %   | 12.503 % | 6.341 %   | 7.976 % |
| Beam Search with Golos train set LM | 3.548 %   | 12.384 % |  -        | -       |
| Beam Search with Common Crawl and Golos LM | 3.318 %   | 11.488 % | 6.4 %     | 8.06 %   |


<sup>1</sup> [Common Voice](https://commonvoice.mozilla.org) - Mozilla's initiative to help teach machines how real people speak.

##  **Resources**

[[arxiv.org] Golos: Russian Dataset for Speech Research](https://arxiv.org/abs/2106.10161)

[[habr.com] Golos — самый большой русскоязычный речевой датасет, размеченный вручную, теперь в открытом доступе](https://habr.com/ru/company/sberdevices/blog/559496/)

[[habr.com] Как улучшить распознавание русской речи до 3% WER с помощью открытых данных](https://habr.com/ru/company/sberdevices/blog/569082/)

arxiv:1910.10261 arxiv:2106.10161 region:us
{}
TAGS
#arxiv-1910.10261 #arxiv-2106.10161 #region-us 

Golos dataset
=============


Golos is a Russian corpus suitable for speech research. The dataset mainly consists of recorded audio files manually annotated on the crowd-sourcing platform. The total duration of the audio is about 1240 hours.
We have made the corpus freely available for downloading, along with the acoustic model prepared on this corpus.
Also we create 3-gram KenLM language model using an open Common Crawl corpus.


Dataset structure
-----------------



Downloads
---------


### Audio files in opus format



### Audio files in wav format


Manifest files with all the training transcription texts are in the train\_crowd9.tar archive listed in the table:


Archives: train\_farfield.tar, Size: 15.4 GB, Links: URL
Archives: train\_crowd0.tar, Size: 11 GB, Links: URL
Archives: train\_crowd1.tar, Size: 14 GB, Links: URL
Archives: train\_crowd2.tar, Size: 13.2 GB, Links: URL
Archives: train\_crowd3.tar, Size: 11.6 GB, Links: URL
Archives: train\_crowd4.tar, Size: 15.8 GB, Links: URL
Archives: train\_crowd5.tar, Size: 13.1 GB, Links: URL
Archives: train\_crowd6.tar, Size: 15.7 GB, Links: URL
Archives: train\_crowd7.tar, Size: 12.7 GB, Links: URL
Archives: train\_crowd8.tar, Size: 12.2 GB, Links: URL
Archives: train\_crowd9.tar, Size: 8.08 GB, Links: URL
Archives: URL, Size: 1.3 GB, Links: URL


### Acoustic and language models


Acoustic model built using QuartzNet15x5 architecture and trained using NeMo toolkit


Three n-gram language models created using KenLM Language Model Toolkit


* LM built on Common Crawl Russian dataset
* LM built on Golos train set
* LM built on Common Crawl and Golos datasets together (50/50)


Archives: QuartzNet15x5\_golos.nemo, Size: 68 MB, Links: URL
Archives: URL, Size: 4.8 GB, Links: URL


Golos data and models are also available in the hub of pre-trained models, datasets, and containers - DataHub ML Space. You can train the model and deploy it on the high-performance SberCloud infrastructure in ML Space - full-cycle machine learning development platform for DS-teams collaboration based on the Christofari Supercomputer.


Evaluation
----------


Percents of Word Error Rate for different test sets



1 Common Voice - Mozilla's initiative to help teach machines how real people speak.


Resources
---------


[[URL] Golos: Russian Dataset for Speech Research](URL


[[URL] Golos — самый большой русскоязычный речевой датасет, размеченный вручную, теперь в открытом доступе](URL


[[URL] Как улучшить распознавание русской речи до 3% WER с помощью открытых данных](URL



### Audio files in opus format ### Audio files in wav format


Manifest files with all the training transcription texts are in the train\_crowd9.tar archive listed in the table:


Archives: train\_farfield.tar, Size: 15.4 GB, Links: URL
Archives: train\_crowd0.tar, Size: 11 GB, Links: URL
Archives: train\_crowd1.tar, Size: 14 GB, Links: URL
Archives: train\_crowd2.tar, Size: 13.2 GB, Links: URL
Archives: train\_crowd3.tar, Size: 11.6 GB, Links: URL
Archives: train\_crowd4.tar, Size: 15.8 GB, Links: URL
Archives: train\_crowd5.tar, Size: 13.1 GB, Links: URL
Archives: train\_crowd6.tar, Size: 15.7 GB, Links: URL
Archives: train\_crowd7.tar, Size: 12.7 GB, Links: URL
Archives: train\_crowd8.tar, Size: 12.2 GB, Links: URL
Archives: train\_crowd9.tar, Size: 8.08 GB, Links: URL
Archives: URL, Size: 1.3 GB, Links: URL ### Acoustic and language models


Acoustic model built using QuartzNet15x5 architecture and trained using NeMo toolkit


Three n-gram language models created using KenLM Language Model Toolkit


* LM built on Common Crawl Russian dataset
* LM built on Golos train set
* LM built on Common Crawl and Golos datasets together (50/50)


Archives: QuartzNet15x5\_golos.nemo, Size: 68 MB, Links: URL
Archives: URL, Size: 4.8 GB, Links: URL


Golos data and models are also available in the hub of pre-trained models, datasets, and containers - DataHub ML Space. You can train the model and deploy it on the high-performance SberCloud infrastructure in ML Space - full-cycle machine learning development platform for DS-teams collaboration based on the Christofari Supercomputer.


Evaluation
----------


Percents of Word Error Rate for different test sets



1 Common Voice - Mozilla's initiative to help teach machines how real people speak.


Resources
---------


[[URL] Golos: Russian Dataset for Speech Research](URL


[[URL] Golos — самый большой русскоязычный речевой датасет, размеченный вручную, теперь в открытом доступе](URL


[[URL] Как улучшить распознавание русской речи до 3% WER с помощью открытых данных](URL
TAGS
#arxiv-1910.10261 #arxiv-2106.10161 #region-us 
 ### Audio files in opus format ### Audio files in wav format


Manifest files with all the training transcription texts are in the train\_crowd9.tar archive listed in the table:


Archives: train\_farfield.tar, Size: 15.4 GB, Links: URL
Archives: train\_crowd0.tar, Size: 11 GB, Links: URL
Archives: train\_crowd1.tar, Size: 14 GB, Links: URL
Archives: train\_crowd2.tar, Size: 13.2 GB, Links: URL
Archives: train\_crowd3.tar, Size: 11.6 GB, Links: URL
Archives: train\_crowd4.tar, Size: 15.8 GB, Links: URL
Archives: train\_crowd5.tar, Size: 13.1 GB, Links: URL
Archives: train\_crowd6.tar, Size: 15.7 GB, Links: URL
Archives: train\_crowd7.tar, Size: 12.7 GB, Links: URL
Archives: train\_crowd8.tar, Size: 12.2 GB, Links: URL
Archives: train\_crowd9.tar, Size: 8.08 GB, Links: URL
Archives: URL, Size: 1.3 GB, Links: URL ### Acoustic and language models


Acoustic model built using QuartzNet15x5 architecture and trained using NeMo toolkit


Three n-gram language models created using KenLM Language Model Toolkit


* LM built on Common Crawl Russian dataset
* LM built on Golos train set
* LM built on Common Crawl and Golos datasets together (50/50)


Archives: QuartzNet15x5\_golos.nemo, Size: 68 MB, Links: URL
Archives: URL, Size: 4.8 GB, Links: URL


Golos data and models are also available in the hub of pre-trained models, datasets, and containers - DataHub ML Space. You can train the model and deploy it on the high-performance SberCloud infrastructure in ML Space - full-cycle machine learning development platform for DS-teams collaboration based on the Christofari Supercomputer.


Evaluation
----------


Percents of Word Error Rate for different test sets



1 Common Voice - Mozilla's initiative to help teach machines how real people speak.


Resources
---------


[[URL] Golos: Russian Dataset for Speech Research](URL


[[URL] Golos — самый большой русскоязычный речевой датасет, размеченный вручную, теперь в открытом доступе](URL


[[URL] Как улучшить распознавание русской речи до 3% WER с помощью открытых данных](URL
passage: TAGS
#arxiv-1910.10261 #arxiv-2106.10161 #region-us 
### Audio files in opus format### Audio files in wav format


Manifest files with all the training transcription texts are in the train\_crowd9.tar archive listed in the table:


Archives: train\_farfield.tar, Size: 15.4 GB, Links: URL
Archives: train\_crowd0.tar, Size: 11 GB, Links: URL
Archives: train\_crowd1.tar, Size: 14 GB, Links: URL
Archives: train\_crowd2.tar, Size: 13.2 GB, Links: URL
Archives: train\_crowd3.tar, Size: 11.6 GB, Links: URL
Archives: train\_crowd4.tar, Size: 15.8 GB, Links: URL
Archives: train\_crowd5.tar, Size: 13.1 GB, Links: URL
Archives: train\_crowd6.tar, Size: 15.7 GB, Links: URL
Archives: train\_crowd7.tar, Size: 12.7 GB, Links: URL
Archives: train\_crowd8.tar, Size: 12.2 GB, Links: URL
Archives: train\_crowd9.tar, Size: 8.08 GB, Links: URL
Archives: URL, Size: 1.3 GB, Links: URL