
# Dataset Card for common_language

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://zenodo.org/record/5036977
- **Repository:** https://github.com/speechbrain/speechbrain/tree/develop/recipes/CommonLanguage
- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Leaderboard:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Dataset Summary

This dataset is composed of speech recordings from languages that were carefully selected from the CommonVoice database. The total duration of audio recordings is 45.1 hours (i.e., 1 hour of material for each language). The dataset has been extracted from CommonVoice to train language-id systems.

### Supported Tasks and Leaderboards

The baselines for language-id are available in the SpeechBrain toolkit (see recipes/CommonLanguage):
https://github.com/speechbrain/speechbrain

### Languages

List of included language:
```
Arabic, Basque, Breton, Catalan, Chinese_China, Chinese_Hongkong, Chinese_Taiwan, Chuvash, Czech, Dhivehi, Dutch, English, Esperanto, Estonian, French, Frisian, Georgian, German, Greek, Hakha_Chin, Indonesian, Interlingua, Italian, Japanese, Kabyle, Kinyarwanda, Kyrgyz, Latvian, Maltese, Mongolian, Persian, Polish, Portuguese, Romanian, Romansh_Sursilvan, Russian, Sakha, Slovenian, Spanish, Swedish, Tamil, Tatar, Turkish, Ukranian, Welsh
```

## Dataset Structure

### Data Instances

A typical data point comprises the `path` to the audio file, and its label `language`. Additional fields include `age`, `client_id`, `gender` and `sentence`.

```python
{
  'client_id': 'itln_trn_sp_175',
  'path': '/path/common_voice_kpd/Italian/train/itln_trn_sp_175/common_voice_it_18279446.wav',
  'sentence': 'Con gli studenti è leggermente simile.',
  'age': 'not_defined',
  'gender': 'not_defined',
  'language': 22
}
```

### Data Fields

`client_id` (`string`): An id for which client (voice) made the recording

`path` (`string`): The path to the audio file

`language` (`ClassLabel`): The language of the recording (see the `Languages` section above)

`sentence` (`string`): The sentence the user was prompted to speak

`age` (`string`): The age of the speaker.

`gender` (`string`): The gender of the speaker

### Data Splits

The dataset is already balanced and split into train, dev (validation) and test sets. 

| Name                              | Train  | Dev    | Test  |
|:---------------------------------:|:------:|:------:|:-----:|
| **# of utterances**               | 177552 | 47104  | 47704 |
| **# unique speakers**             | 11189  | 1297   | 1322  |
| **Total duration, hr**            | 30.04  | 7.53   | 7.53  |
| **Min duration, sec**             | 0.86   | 0.98   | 0.89  |
| **Mean duration, sec**            | 4.87   | 4.61   | 4.55  |
| **Max duration, sec**             | 21.72  | 105.67 | 29.83 |
| **Duration per language, min**    | ~40    | ~10    | ~10   |

## Dataset Creation

### Curation Rationale

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the source language producers?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Annotations

#### Annotation process

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the annotators?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Personal and Sensitive Information

The dataset consists of people who have donated their voice online.  You agree to not attempt to determine the identity of speakers in the Common Voice dataset.

## Considerations for Using the Data

### Social Impact of Dataset

The dataset consists of people who have donated their voice online.  You agree to not attempt to determine the identity of speakers in the Common Voice dataset.

### Discussion of Biases

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Other Known Limitations

The Mongolian and Ukrainian languages are spelled as "Mangolian" and "Ukranian" in this version of the dataset.

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Additional Information

### Dataset Curators

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Licensing Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Citation Information

```
@dataset{ganesh_sinisetty_2021_5036977,
  author       = {Ganesh Sinisetty and
                  Pavlo Ruban and
                  Oleksandr Dymov and
                  Mirco Ravanelli},
  title        = {CommonLanguage},
  month        = jun,
  year         = 2021,
  publisher    = {Zenodo},
  version      = {0.1},
  doi          = {10.5281/zenodo.5036977},
  url          = {https://doi.org/10.5281/zenodo.5036977}
}
```

### Contributions

Thanks to [@anton-l](https://github.com/anton-l) for adding this dataset.

annotations_creators:crowdsourced language_creators:crowdsourced multilinguality:multilingual size_categories:100K<n<1M source_datasets:extended|common_voice language:ar language:br language:ca language:cnh language:cs language:cv language:cy language:de language:dv language:el language:en language:eo language:es language:et language:eu language:fa language:fr language:fy language:ia language:id language:it language:ja language:ka language:kab language:ky language:lv language:mn language:mt language:nl language:pl language:pt language:rm language:ro language:ru language:rw language:sah language:sl language:sv language:ta language:tr language:tt language:uk language:zh license:cc-by-nc-4.0 region:us
{"annotations_creators": ["crowdsourced"], "language_creators": ["crowdsourced"], "language": ["ar", "br", "ca", "cnh", "cs", "cv", "cy", "de", "dv", "el", "en", "eo", "es", "et", "eu", "fa", "fr", "fy", "ia", "id", "it", "ja", "ka", "kab", "ky", "lv", "mn", "mt", "nl", "pl", "pt", "rm", "ro", "ru", "rw", "sah", "sl", "sv", "ta", "tr", "tt", "uk", "zh"], "license": ["cc-by-nc-4.0"], "multilinguality": ["multilingual"], "size_categories": ["100K<n<1M"], "source_datasets": ["extended|common_voice"], "task_categories": ["speech-processing"], "task_ids": ["speech-classification"], "pretty_name": "Common Language", "language_bcp47": ["ar", "br", "ca", "cnh", "cs", "cv", "cy", "de", "dv", "el", "en", "eo", "es", "et", "eu", "fa", "fr", "fy-NL", "ia", "id", "it", "ja", "ka", "kab", "ky", "lv", "mn", "mt", "nl", "pl", "pt", "rm-sursilv", "ro", "ru", "rw", "sah", "sl", "sv-SE", "ta", "tr", "tt", "uk", "zh-CN", "zh-HK", "zh-TW"]}
TAGS
#annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-extended|common_voice #language-Arabic #language-Breton #language-Catalan #language-Hakha Chin #language-Czech #language-Chuvash #language-Welsh #language-German #language-Dhivehi #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-French #language-Western Frisian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Italian #language-Japanese #language-Georgian #language-Kabyle #language-Kirghiz #language-Latvian #language-Mongolian #language-Maltese #language-Dutch #language-Polish #language-Portuguese #language-Romansh #language-Romanian #language-Russian #language-Kinyarwanda #language-Yakut #language-Slovenian #language-Swedish #language-Tamil #language-Turkish #language-Tatar #language-Ukrainian #language-Chinese #license-cc-by-nc-4.0 #region-us 

Dataset Card for common\_language
=================================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper:
* Leaderboard:
* Point of Contact:


### Dataset Summary


This dataset is composed of speech recordings from languages that were carefully selected from the CommonVoice database. The total duration of audio recordings is 45.1 hours (i.e., 1 hour of material for each language). The dataset has been extracted from CommonVoice to train language-id systems.


### Supported Tasks and Leaderboards


The baselines for language-id are available in the SpeechBrain toolkit (see recipes/CommonLanguage):
URL


### Languages


List of included language:


Dataset Structure
-----------------


### Data Instances


A typical data point comprises the 'path' to the audio file, and its label 'language'. Additional fields include 'age', 'client\_id', 'gender' and 'sentence'.


### Data Fields


'client\_id' ('string'): An id for which client (voice) made the recording


'path' ('string'): The path to the audio file


'language' ('ClassLabel'): The language of the recording (see the 'Languages' section above)


'sentence' ('string'): The sentence the user was prompted to speak


'age' ('string'): The age of the speaker.


'gender' ('string'): The gender of the speaker


### Data Splits


The dataset is already balanced and split into train, dev (validation) and test sets.



Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


The dataset consists of people who have donated their voice online. You agree to not attempt to determine the identity of speakers in the Common Voice dataset.


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


The dataset consists of people who have donated their voice online. You agree to not attempt to determine the identity of speakers in the Common Voice dataset.


### Discussion of Biases


### Other Known Limitations


The Mongolian and Ukrainian languages are spelled as "Mangolian" and "Ukranian" in this version of the dataset.


Additional Information
----------------------


### Dataset Curators


### Licensing Information


### Contributions


Thanks to @anton-l for adding this dataset.



### Dataset Summary


This dataset is composed of speech recordings from languages that were carefully selected from the CommonVoice database. The total duration of audio recordings is 45.1 hours (i.e., 1 hour of material for each language). The dataset has been extracted from CommonVoice to train language-id systems. ### Supported Tasks and Leaderboards


The baselines for language-id are available in the SpeechBrain toolkit (see recipes/CommonLanguage):
URL ### Languages


List of included language:


Dataset Structure
----------------- ### Data Instances


A typical data point comprises the 'path' to the audio file, and its label 'language'. Additional fields include 'age', 'client\_id', 'gender' and 'sentence'. ### Data Fields


'client\_id' ('string'): An id for which client (voice) made the recording


'path' ('string'): The path to the audio file


'language' ('ClassLabel'): The language of the recording (see the 'Languages' section above)


'sentence' ('string'): The sentence the user was prompted to speak


'age' ('string'): The age of the speaker.


'gender' ('string'): The gender of the speaker ### Data Splits


The dataset is already balanced and split into train, dev (validation) and test sets.



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


The dataset consists of people who have donated their voice online. You agree to not attempt to determine the identity of speakers in the Common Voice dataset.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


The dataset consists of people who have donated their voice online. You agree to not attempt to determine the identity of speakers in the Common Voice dataset. ### Discussion of Biases ### Other Known Limitations


The Mongolian and Ukrainian languages are spelled as "Mangolian" and "Ukranian" in this version of the dataset.


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions


Thanks to @anton-l for adding this dataset.
TAGS
#annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-extended|common_voice #language-Arabic #language-Breton #language-Catalan #language-Hakha Chin #language-Czech #language-Chuvash #language-Welsh #language-German #language-Dhivehi #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-French #language-Western Frisian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Italian #language-Japanese #language-Georgian #language-Kabyle #language-Kirghiz #language-Latvian #language-Mongolian #language-Maltese #language-Dutch #language-Polish #language-Portuguese #language-Romansh #language-Romanian #language-Russian #language-Kinyarwanda #language-Yakut #language-Slovenian #language-Swedish #language-Tamil #language-Turkish #language-Tatar #language-Ukrainian #language-Chinese #license-cc-by-nc-4.0 #region-us 
 ### Dataset Summary


This dataset is composed of speech recordings from languages that were carefully selected from the CommonVoice database. The total duration of audio recordings is 45.1 hours (i.e., 1 hour of material for each language). The dataset has been extracted from CommonVoice to train language-id systems. ### Supported Tasks and Leaderboards


The baselines for language-id are available in the SpeechBrain toolkit (see recipes/CommonLanguage):
URL ### Languages


List of included language:


Dataset Structure
----------------- ### Data Instances


A typical data point comprises the 'path' to the audio file, and its label 'language'. Additional fields include 'age', 'client\_id', 'gender' and 'sentence'. ### Data Fields


'client\_id' ('string'): An id for which client (voice) made the recording


'path' ('string'): The path to the audio file


'language' ('ClassLabel'): The language of the recording (see the 'Languages' section above)


'sentence' ('string'): The sentence the user was prompted to speak


'age' ('string'): The age of the speaker.


'gender' ('string'): The gender of the speaker ### Data Splits


The dataset is already balanced and split into train, dev (validation) and test sets.



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


The dataset consists of people who have donated their voice online. You agree to not attempt to determine the identity of speakers in the Common Voice dataset.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


The dataset consists of people who have donated their voice online. You agree to not attempt to determine the identity of speakers in the Common Voice dataset. ### Discussion of Biases ### Other Known Limitations


The Mongolian and Ukrainian languages are spelled as "Mangolian" and "Ukranian" in this version of the dataset.


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions


Thanks to @anton-l for adding this dataset.
passage: TAGS
#annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-extended|common_voice #language-Arabic #language-Breton #language-Catalan #language-Hakha Chin #language-Czech #language-Chuvash #language-Welsh #language-German #language-Dhivehi #language-Modern Greek (1453-) #language-English #language-Esperanto #language-Spanish #language-Estonian #language-Basque #language-Persian #language-French #language-Western Frisian #language-Interlingua (International Auxiliary Language Association) #language-Indonesian #language-Italian #language-Japanese #language-Georgian #language-Kabyle #language-Kirghiz #language-Latvian #language-Mongolian #language-Maltese #language-Dutch #language-Polish #language-Portuguese #language-Romansh #language-Romanian #language-Russian #language-Kinyarwanda #language-Yakut #language-Slovenian #language-Swedish #language-Tamil #language-Turkish #language-Tatar #language-Ukrainian #language-Chinese #license-cc-by-nc-4.0 #region-us 
### Dataset Summary


This dataset is composed of speech recordings from languages that were carefully selected from the CommonVoice database. The total duration of audio recordings is 45.1 hours (i.e., 1 hour of material for each language). The dataset has been extracted from CommonVoice to train language-id systems.### Supported Tasks and Leaderboards


The baselines for language-id are available in the SpeechBrain toolkit (see recipes/CommonLanguage):
URL### Languages


List of included language:


Dataset Structure
-----------------