

# MOCKS: Multilingual Open Custom Keyword Spotting Testset

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Paper:** [MOCKS 1.0: Multilingual Open Custom Keyword Spotting Testset](https://www.isca-speech.org/archive/pdfs/interspeech_2023/pudo23_interspeech.pdf)

### Dataset Summary

Multilingual Open Custom Keyword Spotting Testset (MOCKS) is a comprehensive audio testset for evaluation and benchmarking
Open-Vocabulary Keyword Spotting (OV-KWS) models. It supports multiple OV-KWS problems:
both text-based and audio-based keyword spotting, as well as offline and online (streaming) modes. 
It is based on the LibriSpeech and Mozilla Common Voice datasets and contains
almost 50,000 keywords, with audio data available in English, French, German, Italian, and Spanish.
The testset was generated using automatically generated alignments used for the extraction of parts of the recordings that were split into keywords and test samples.
MOCKS contains both positive and negative examples selected based on phonetic transcriptions that are challenging and should allow for in-depth OV-KWS model evaluation. 

Please refer to our [paper](https://www.isca-speech.org/archive/pdfs/interspeech_2023/pudo23_interspeech.pdf) for further details.

### Supported Tasks and Leaderboards

The MOCKS dataset can be used for the Open-Vocabulary Keyword Spotting (OV-KWS) task. It supports two OV-KWS types:
- Query-by-Text, where the keyword is provided by text and needs to be detected in the audio stream.
- Query-by-Example, where the keyword is provided with enrollment audio for detection in the audio stream.

It also allows for:
- offline keyword detection, where test audio is trimmed to contain only keywords of interest.
- online (streaming) keyword detection, where test audio has past and future context besides keywords of interest.

### Languages

The MOCKS incorporates 5 languages:
- English - primary and largest test set,
- German,
- Spanish,
- French,
- Italian.

## Dataset Structure

The MOCKS testset is split by language, source dataset, and OV-KWS type:
```
MOCKS
│
└───de
│   └───MCV
│   │   └───test
│   │   │   └───offline
│   │   │   │   │   all.pair.different.tsv
│   │   │   │   │   all.pair.positive.tsv
│   │   │   │   │   all.pair.similar.tsv
│   │   │   │   │   data.tar.gz
│   │   │   │   │   subset.pair.different.tsv
│   │   │   │   │   subset.pair.positive.tsv
│   │   │   │   │   subset.pair.similar.tsv
│   │   │   │
│   │   │   └───online
│   │   │   │   │   all.pair.different.tsv
│   │   │   │   │   ...
│   │   │   │   data.offline.transcription.tsv
│   │   │   │   data.online.transcription.tsv
│
└───en
│   └───LS-clean
│   │   └───test
│   │   │   └───offline
│   │   │   │   │   all.pair.different.tsv
│   │   │   │   │   ...
│   │   │   │   ...
│   │
│   └───LS-other
│   │   └───test
│   │   │   └───offline
│   │   │   │   │   all.pair.different.tsv
│   │   │   │   │   ...
│   │   │   │   ...
│   │
│   └───MCV
│   │   └───test
│   │   │   └───offline
│   │   │   │   │   all.pair.different.tsv
│   │   │   │   │   ...
│   │   │   │   ...
│
└───...
```

Each split is divided into:
- positive examples (`all.pair.positive.tsv`) - test examples with true keywords, 5000-8000 keywords in each subset,
- similar examples (`all.pair.similar.tsv`) - test examples with similar phrases to the keyword selected based on phonetic transcription distance,
- different examples (`all.pair.different.tsv`) - test examples with completely different phrases.

All those files contain columns separated by tab:
- `keyword_path` - path to audio containing keyword phrase.
- `adversary_keyword_path` - path to test audio.
- `adversary_keyword_timestamp_start` - start time in seconds of phrase of interest for a given keyword from `keyword_path`, the field only available in **offline** split.
- `adversary_keyword_timestamp_end` - end time in seconds of phrase of interest for a given keyword from `keyword_path`, the field only available in **offline** split.
- `label` - whether the `adversary_keyword_path` contain keyword from `keyword_path` or not (1 - contains keyword, 0 - doesn't contain keyword).

Each split also contains a subset of whole data with the same field structure to allow faster evaluation (`subset.pair.*.tsv`).

Also, transcriptions are provided for each audio in:
- `data_offline_transcription.tsv` - transcriptions for **offline** examples and `keyword_path` from **online** scenario,
- `data_online_transcription.tsv` - transcriptions for the adversary, test examples from **online** scenario,

three columns are present within each file:
- `path_to_keyword`/`path_to_adversary_keyword` - path to the audio file,
- `keyword_transcription`/`adversary_keyword_transcription` - audio transcription,
- `keyword_phonetic_transcription`/`adversary_keyword_phonetic_transcription` - audio phonetic transcription.

## Using the Dataset

The dataset can be used by:
- downloading the archive and constructing all the test cases based on the provided `tsv` files,
- `datasets` package.

In the latter case, the following should work:
```
load_dataset(path="voiceintelligenceresearch/MOCKS", name="en.LS-clean", split="offline")
```

The allowed values for `name` are:
- `en.LS-{clean,other}`,
- `en.LS-{clean,other}.positive`,
- `en.LS-{clean,other}.similar`,
- `en.LS-{clean,other}.different`,
- `en.LS-{clean,other}.subset`,
- `en.LS-{clean,other}.positive_subset`,
- `en.LS-{clean,other}.similar_subset`,
- `en.LS-{clean,other}.different_subset`,
- `{de,en,es,fr,it}.MCV.positive`,
- `{de,en,es,fr,it}.MCV.positive.similar`,
- `{de,en,es,fr,it}.MCV.positive.different`,
- `{de,en,es,fr,it}.MCV.positive.subset`,
- `{de,en,es,fr,it}.MCV.positive.positive_subset`,
- `{de,en,es,fr,it}.MCV.positive.similar_subset`,
- `{de,en,es,fr,it}.MCV.positive.different_subset`.

The allowed values for `split` are:
- `offline`,
- `online`.

`load_dataset` provides a list of the dictionary objects with the following contents:
```
{
  "keyword_id": datasets.Value("string"),
  "keyword_transcription": datasets.Value("string"),
  "test_id": datasets.Value("string"),
  "test_transcription": datasets.Value("string"),
  "test_audio": datasets.Audio(sampling_rate=16000),
  "label": datasets.Value("bool"),
}
```

Each element of this list represents a single test case for the QbyT KWS:
- `keyword_id` - the name of the keyword audio file in `data.tar.gz` (not used in QbyT KWS),
- `keyword_transcription` - transcription of the keyword,
- `test_id` - the name of the test audio file in `data.tar.gz`,
- `test_transcription` - transcription of the test sample,
- `test_audio` - raw data of the test audio,
- `label` - `True` if the test case is positive (`keyword_transcription` is a substring of the `test_transcription`), `False` otherwise (`similar` and `different` subsets).

Note that each test case can be extended to QbyE KWS by reading the proper `keyword_id` file. Unfortunately, there is no easy way to do that in the loading script.

All the test files are provided in 16 kHz, even though `{de,en,es,fr,it}.MCV` files are stored in the original sampling (usually 48 kHz) in the `data.tar.gz` archives.

## Dataset Creation

The MOCKS testset was created from LibriSpeech and Mozilla Common Voice (MCV) datasets that are publicly available. To create it:
- a [MFA](https://mfa-models.readthedocs.io/en/latest/acoustic/index.html) with publicly available models was used to extract word-level alignments,
- an internally developed, rule-based grapheme-to-phoneme (G2P) algorithm was used to prepare phonetic transcriptions for each sample.

The data is stored in a 16-bit, single-channel WAV format. 16kHz sampling rate is used for LibriSpeech based testset
and 48kHz sampling rate for MCV based testset.

The offline testset contains an additional 0.1 seconds at the beginning and end of the extracted audio sample to mitigate the cut-speech effect.
The online version contains an additional 1 second or so at the beginning and end of the extracted audio sample.

The MOCKS testset is gender balanced.

## Citation Information

```bibtex
@inproceedings{pudo23_interspeech,
  author={Mikołaj Pudo and Mateusz Wosik and Adam Cieślak and Justyna Krzywdziak and Bożena Łukasiak and Artur Janicki},
  title={{MOCKS} 1.0: Multilingual Open Custom Keyword Spotting Testset},
  year={2023},
  booktitle={Proc. Interspeech 2023},
}
```
annotations_creators:expert-generated multilinguality:multilingual language:en language:de language:es language:fr language:it license:cc-by-4.0 license:mpl-2.0 region:us
{"annotations_creators": ["expert-generated"], "language": ["en", "de", "es", "fr", "it"], "license": ["cc-by-4.0", "mpl-2.0"], "multilinguality": ["multilingual"], "dataset_info": [{"config_name": "config", "features": [{"name": "audio_id", "dtype": "string"}, {"name": "audio", "dtype": {"audio": {"sampling_rate": 16000}}}, {"name": "text", "dtype": "string"}]}]}
TAGS
#annotations_creators-expert-generated #multilinguality-multilingual #language-English #language-German #language-Spanish #language-French #language-Italian #license-cc-by-4.0 #license-mpl-2.0 #region-us 



# MOCKS: Multilingual Open Custom Keyword Spotting Testset

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Paper: MOCKS 1.0: Multilingual Open Custom Keyword Spotting Testset

### Dataset Summary

Multilingual Open Custom Keyword Spotting Testset (MOCKS) is a comprehensive audio testset for evaluation and benchmarking
Open-Vocabulary Keyword Spotting (OV-KWS) models. It supports multiple OV-KWS problems:
both text-based and audio-based keyword spotting, as well as offline and online (streaming) modes. 
It is based on the LibriSpeech and Mozilla Common Voice datasets and contains
almost 50,000 keywords, with audio data available in English, French, German, Italian, and Spanish.
The testset was generated using automatically generated alignments used for the extraction of parts of the recordings that were split into keywords and test samples.
MOCKS contains both positive and negative examples selected based on phonetic transcriptions that are challenging and should allow for in-depth OV-KWS model evaluation. 

Please refer to our paper for further details.

### Supported Tasks and Leaderboards

The MOCKS dataset can be used for the Open-Vocabulary Keyword Spotting (OV-KWS) task. It supports two OV-KWS types:
- Query-by-Text, where the keyword is provided by text and needs to be detected in the audio stream.
- Query-by-Example, where the keyword is provided with enrollment audio for detection in the audio stream.

It also allows for:
- offline keyword detection, where test audio is trimmed to contain only keywords of interest.
- online (streaming) keyword detection, where test audio has past and future context besides keywords of interest.

### Languages

The MOCKS incorporates 5 languages:
- English - primary and largest test set,
- German,
- Spanish,
- French,
- Italian.

## Dataset Structure

The MOCKS testset is split by language, source dataset, and OV-KWS type:


Each split is divided into:
- positive examples ('URL') - test examples with true keywords, 5000-8000 keywords in each subset,
- similar examples ('URL') - test examples with similar phrases to the keyword selected based on phonetic transcription distance,
- different examples ('URL') - test examples with completely different phrases.

All those files contain columns separated by tab:
- 'keyword_path' - path to audio containing keyword phrase.
- 'adversary_keyword_path' - path to test audio.
- 'adversary_keyword_timestamp_start' - start time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'adversary_keyword_timestamp_end' - end time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'label' - whether the 'adversary_keyword_path' contain keyword from 'keyword_path' or not (1 - contains keyword, 0 - doesn't contain keyword).

Each split also contains a subset of whole data with the same field structure to allow faster evaluation ('URL.*.tsv').

Also, transcriptions are provided for each audio in:
- 'data_offline_transcription.tsv' - transcriptions for offline examples and 'keyword_path' from online scenario,
- 'data_online_transcription.tsv' - transcriptions for the adversary, test examples from online scenario,

three columns are present within each file:
- 'path_to_keyword'/'path_to_adversary_keyword' - path to the audio file,
- 'keyword_transcription'/'adversary_keyword_transcription' - audio transcription,
- 'keyword_phonetic_transcription'/'adversary_keyword_phonetic_transcription' - audio phonetic transcription.

## Using the Dataset

The dataset can be used by:
- downloading the archive and constructing all the test cases based on the provided 'tsv' files,
- 'datasets' package.

In the latter case, the following should work:


The allowed values for 'name' are:
- 'en.LS-{clean,other}',
- 'en.LS-{clean,other}.positive',
- 'en.LS-{clean,other}.similar',
- 'en.LS-{clean,other}.different',
- 'en.LS-{clean,other}.subset',
- 'en.LS-{clean,other}.positive_subset',
- 'en.LS-{clean,other}.similar_subset',
- 'en.LS-{clean,other}.different_subset',
- '{de,en,es,fr,it}.MCV.positive',
- '{de,en,es,fr,it}.MCV.positive.similar',
- '{de,en,es,fr,it}.MCV.positive.different',
- '{de,en,es,fr,it}.URL',
- '{de,en,es,fr,it}.MCV.positive.positive_subset',
- '{de,en,es,fr,it}.MCV.positive.similar_subset',
- '{de,en,es,fr,it}.MCV.positive.different_subset'.

The allowed values for 'split' are:
- 'offline',
- 'online'.

'load_dataset' provides a list of the dictionary objects with the following contents:


Each element of this list represents a single test case for the QbyT KWS:
- 'keyword_id' - the name of the keyword audio file in 'URL' (not used in QbyT KWS),
- 'keyword_transcription' - transcription of the keyword,
- 'test_id' - the name of the test audio file in 'URL',
- 'test_transcription' - transcription of the test sample,
- 'test_audio' - raw data of the test audio,
- 'label' - 'True' if the test case is positive ('keyword_transcription' is a substring of the 'test_transcription'), 'False' otherwise ('similar' and 'different' subsets).

Note that each test case can be extended to QbyE KWS by reading the proper 'keyword_id' file. Unfortunately, there is no easy way to do that in the loading script.

All the test files are provided in 16 kHz, even though '{de,en,es,fr,it}.MCV' files are stored in the original sampling (usually 48 kHz) in the 'URL' archives.

## Dataset Creation

The MOCKS testset was created from LibriSpeech and Mozilla Common Voice (MCV) datasets that are publicly available. To create it:
- a MFA with publicly available models was used to extract word-level alignments,
- an internally developed, rule-based grapheme-to-phoneme (G2P) algorithm was used to prepare phonetic transcriptions for each sample.

The data is stored in a 16-bit, single-channel WAV format. 16kHz sampling rate is used for LibriSpeech based testset
and 48kHz sampling rate for MCV based testset.

The offline testset contains an additional 0.1 seconds at the beginning and end of the extracted audio sample to mitigate the cut-speech effect.
The online version contains an additional 1 second or so at the beginning and end of the extracted audio sample.

The MOCKS testset is gender balanced.




# MOCKS: Multilingual Open Custom Keyword Spotting Testset ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Paper: MOCKS 1.0: Multilingual Open Custom Keyword Spotting Testset ### Dataset Summary

Multilingual Open Custom Keyword Spotting Testset (MOCKS) is a comprehensive audio testset for evaluation and benchmarking
Open-Vocabulary Keyword Spotting (OV-KWS) models. It supports multiple OV-KWS problems:
both text-based and audio-based keyword spotting, as well as offline and online (streaming) modes. 
It is based on the LibriSpeech and Mozilla Common Voice datasets and contains
almost 50,000 keywords, with audio data available in English, French, German, Italian, and Spanish.
The testset was generated using automatically generated alignments used for the extraction of parts of the recordings that were split into keywords and test samples.
MOCKS contains both positive and negative examples selected based on phonetic transcriptions that are challenging and should allow for in-depth OV-KWS model evaluation. 

Please refer to our paper for further details. ### Supported Tasks and Leaderboards

The MOCKS dataset can be used for the Open-Vocabulary Keyword Spotting (OV-KWS) task. It supports two OV-KWS types:
- Query-by-Text, where the keyword is provided by text and needs to be detected in the audio stream.
- Query-by-Example, where the keyword is provided with enrollment audio for detection in the audio stream.

It also allows for:
- offline keyword detection, where test audio is trimmed to contain only keywords of interest.
- online (streaming) keyword detection, where test audio has past and future context besides keywords of interest. ### Languages

The MOCKS incorporates 5 languages:
- English - primary and largest test set,
- German,
- Spanish,
- French,
- Italian. ## Dataset Structure

The MOCKS testset is split by language, source dataset, and OV-KWS type:


Each split is divided into:
- positive examples ('URL') - test examples with true keywords, 5000-8000 keywords in each subset,
- similar examples ('URL') - test examples with similar phrases to the keyword selected based on phonetic transcription distance,
- different examples ('URL') - test examples with completely different phrases.

All those files contain columns separated by tab:
- 'keyword_path' - path to audio containing keyword phrase.
- 'adversary_keyword_path' - path to test audio.
- 'adversary_keyword_timestamp_start' - start time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'adversary_keyword_timestamp_end' - end time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'label' - whether the 'adversary_keyword_path' contain keyword from 'keyword_path' or not (1 - contains keyword, 0 - doesn't contain keyword).

Each split also contains a subset of whole data with the same field structure to allow faster evaluation ('URL.*.tsv').

Also, transcriptions are provided for each audio in:
- 'data_offline_transcription.tsv' - transcriptions for offline examples and 'keyword_path' from online scenario,
- 'data_online_transcription.tsv' - transcriptions for the adversary, test examples from online scenario,

three columns are present within each file:
- 'path_to_keyword'/'path_to_adversary_keyword' - path to the audio file,
- 'keyword_transcription'/'adversary_keyword_transcription' - audio transcription,
- 'keyword_phonetic_transcription'/'adversary_keyword_phonetic_transcription' - audio phonetic transcription. ## Using the Dataset

The dataset can be used by:
- downloading the archive and constructing all the test cases based on the provided 'tsv' files,
- 'datasets' package.

In the latter case, the following should work:


The allowed values for 'name' are:
- 'en.LS-{clean,other}',
- 'en.LS-{clean,other}.positive',
- 'en.LS-{clean,other}.similar',
- 'en.LS-{clean,other}.different',
- 'en.LS-{clean,other}.subset',
- 'en.LS-{clean,other}.positive_subset',
- 'en.LS-{clean,other}.similar_subset',
- 'en.LS-{clean,other}.different_subset',
- '{de,en,es,fr,it}.MCV.positive',
- '{de,en,es,fr,it}.MCV.positive.similar',
- '{de,en,es,fr,it}.MCV.positive.different',
- '{de,en,es,fr,it}.URL',
- '{de,en,es,fr,it}.MCV.positive.positive_subset',
- '{de,en,es,fr,it}.MCV.positive.similar_subset',
- '{de,en,es,fr,it}.MCV.positive.different_subset'.

The allowed values for 'split' are:
- 'offline',
- 'online'.

'load_dataset' provides a list of the dictionary objects with the following contents:


Each element of this list represents a single test case for the QbyT KWS:
- 'keyword_id' - the name of the keyword audio file in 'URL' (not used in QbyT KWS),
- 'keyword_transcription' - transcription of the keyword,
- 'test_id' - the name of the test audio file in 'URL',
- 'test_transcription' - transcription of the test sample,
- 'test_audio' - raw data of the test audio,
- 'label' - 'True' if the test case is positive ('keyword_transcription' is a substring of the 'test_transcription'), 'False' otherwise ('similar' and 'different' subsets).

Note that each test case can be extended to QbyE KWS by reading the proper 'keyword_id' file. Unfortunately, there is no easy way to do that in the loading script.

All the test files are provided in 16 kHz, even though '{de,en,es,fr,it}.MCV' files are stored in the original sampling (usually 48 kHz) in the 'URL' archives. ## Dataset Creation

The MOCKS testset was created from LibriSpeech and Mozilla Common Voice (MCV) datasets that are publicly available. To create it:
- a MFA with publicly available models was used to extract word-level alignments,
- an internally developed, rule-based grapheme-to-phoneme (G2P) algorithm was used to prepare phonetic transcriptions for each sample.

The data is stored in a 16-bit, single-channel WAV format. 16kHz sampling rate is used for LibriSpeech based testset
and 48kHz sampling rate for MCV based testset.

The offline testset contains an additional 0.1 seconds at the beginning and end of the extracted audio sample to mitigate the cut-speech effect.
The online version contains an additional 1 second or so at the beginning and end of the extracted audio sample.

The MOCKS testset is gender balanced.
TAGS
#annotations_creators-expert-generated #multilinguality-multilingual #language-English #language-German #language-Spanish #language-French #language-Italian #license-cc-by-4.0 #license-mpl-2.0 #region-us 
 # MOCKS: Multilingual Open Custom Keyword Spotting Testset ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Paper: MOCKS 1.0: Multilingual Open Custom Keyword Spotting Testset ### Dataset Summary

Multilingual Open Custom Keyword Spotting Testset (MOCKS) is a comprehensive audio testset for evaluation and benchmarking
Open-Vocabulary Keyword Spotting (OV-KWS) models. It supports multiple OV-KWS problems:
both text-based and audio-based keyword spotting, as well as offline and online (streaming) modes. 
It is based on the LibriSpeech and Mozilla Common Voice datasets and contains
almost 50,000 keywords, with audio data available in English, French, German, Italian, and Spanish.
The testset was generated using automatically generated alignments used for the extraction of parts of the recordings that were split into keywords and test samples.
MOCKS contains both positive and negative examples selected based on phonetic transcriptions that are challenging and should allow for in-depth OV-KWS model evaluation. 

Please refer to our paper for further details. ### Supported Tasks and Leaderboards

The MOCKS dataset can be used for the Open-Vocabulary Keyword Spotting (OV-KWS) task. It supports two OV-KWS types:
- Query-by-Text, where the keyword is provided by text and needs to be detected in the audio stream.
- Query-by-Example, where the keyword is provided with enrollment audio for detection in the audio stream.

It also allows for:
- offline keyword detection, where test audio is trimmed to contain only keywords of interest.
- online (streaming) keyword detection, where test audio has past and future context besides keywords of interest. ### Languages

The MOCKS incorporates 5 languages:
- English - primary and largest test set,
- German,
- Spanish,
- French,
- Italian. ## Dataset Structure

The MOCKS testset is split by language, source dataset, and OV-KWS type:


Each split is divided into:
- positive examples ('URL') - test examples with true keywords, 5000-8000 keywords in each subset,
- similar examples ('URL') - test examples with similar phrases to the keyword selected based on phonetic transcription distance,
- different examples ('URL') - test examples with completely different phrases.

All those files contain columns separated by tab:
- 'keyword_path' - path to audio containing keyword phrase.
- 'adversary_keyword_path' - path to test audio.
- 'adversary_keyword_timestamp_start' - start time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'adversary_keyword_timestamp_end' - end time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'label' - whether the 'adversary_keyword_path' contain keyword from 'keyword_path' or not (1 - contains keyword, 0 - doesn't contain keyword).

Each split also contains a subset of whole data with the same field structure to allow faster evaluation ('URL.*.tsv').

Also, transcriptions are provided for each audio in:
- 'data_offline_transcription.tsv' - transcriptions for offline examples and 'keyword_path' from online scenario,
- 'data_online_transcription.tsv' - transcriptions for the adversary, test examples from online scenario,

three columns are present within each file:
- 'path_to_keyword'/'path_to_adversary_keyword' - path to the audio file,
- 'keyword_transcription'/'adversary_keyword_transcription' - audio transcription,
- 'keyword_phonetic_transcription'/'adversary_keyword_phonetic_transcription' - audio phonetic transcription. ## Using the Dataset

The dataset can be used by:
- downloading the archive and constructing all the test cases based on the provided 'tsv' files,
- 'datasets' package.

In the latter case, the following should work:


The allowed values for 'name' are:
- 'en.LS-{clean,other}',
- 'en.LS-{clean,other}.positive',
- 'en.LS-{clean,other}.similar',
- 'en.LS-{clean,other}.different',
- 'en.LS-{clean,other}.subset',
- 'en.LS-{clean,other}.positive_subset',
- 'en.LS-{clean,other}.similar_subset',
- 'en.LS-{clean,other}.different_subset',
- '{de,en,es,fr,it}.MCV.positive',
- '{de,en,es,fr,it}.MCV.positive.similar',
- '{de,en,es,fr,it}.MCV.positive.different',
- '{de,en,es,fr,it}.URL',
- '{de,en,es,fr,it}.MCV.positive.positive_subset',
- '{de,en,es,fr,it}.MCV.positive.similar_subset',
- '{de,en,es,fr,it}.MCV.positive.different_subset'.

The allowed values for 'split' are:
- 'offline',
- 'online'.

'load_dataset' provides a list of the dictionary objects with the following contents:


Each element of this list represents a single test case for the QbyT KWS:
- 'keyword_id' - the name of the keyword audio file in 'URL' (not used in QbyT KWS),
- 'keyword_transcription' - transcription of the keyword,
- 'test_id' - the name of the test audio file in 'URL',
- 'test_transcription' - transcription of the test sample,
- 'test_audio' - raw data of the test audio,
- 'label' - 'True' if the test case is positive ('keyword_transcription' is a substring of the 'test_transcription'), 'False' otherwise ('similar' and 'different' subsets).

Note that each test case can be extended to QbyE KWS by reading the proper 'keyword_id' file. Unfortunately, there is no easy way to do that in the loading script.

All the test files are provided in 16 kHz, even though '{de,en,es,fr,it}.MCV' files are stored in the original sampling (usually 48 kHz) in the 'URL' archives. ## Dataset Creation

The MOCKS testset was created from LibriSpeech and Mozilla Common Voice (MCV) datasets that are publicly available. To create it:
- a MFA with publicly available models was used to extract word-level alignments,
- an internally developed, rule-based grapheme-to-phoneme (G2P) algorithm was used to prepare phonetic transcriptions for each sample.

The data is stored in a 16-bit, single-channel WAV format. 16kHz sampling rate is used for LibriSpeech based testset
and 48kHz sampling rate for MCV based testset.

The offline testset contains an additional 0.1 seconds at the beginning and end of the extracted audio sample to mitigate the cut-speech effect.
The online version contains an additional 1 second or so at the beginning and end of the extracted audio sample.

The MOCKS testset is gender balanced.
passage: TAGS
#annotations_creators-expert-generated #multilinguality-multilingual #language-English #language-German #language-Spanish #language-French #language-Italian #license-cc-by-4.0 #license-mpl-2.0 #region-us 
# MOCKS: Multilingual Open Custom Keyword Spotting Testset## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions## Dataset Description

- Paper: MOCKS 1.0: Multilingual Open Custom Keyword Spotting Testset### Dataset Summary

Multilingual Open Custom Keyword Spotting Testset (MOCKS) is a comprehensive audio testset for evaluation and benchmarking
Open-Vocabulary Keyword Spotting (OV-KWS) models. It supports multiple OV-KWS problems:
both text-based and audio-based keyword spotting, as well as offline and online (streaming) modes. 
It is based on the LibriSpeech and Mozilla Common Voice datasets and contains
almost 50,000 keywords, with audio data available in English, French, German, Italian, and Spanish.
The testset was generated using automatically generated alignments used for the extraction of parts of the recordings that were split into keywords and test samples.
MOCKS contains both positive and negative examples selected based on phonetic transcriptions that are challenging and should allow for in-depth OV-KWS model evaluation. 

Please refer to our paper for further details. passage: ### Supported Tasks and Leaderboards

The MOCKS dataset can be used for the Open-Vocabulary Keyword Spotting (OV-KWS) task. It supports two OV-KWS types:
- Query-by-Text, where the keyword is provided by text and needs to be detected in the audio stream.
- Query-by-Example, where the keyword is provided with enrollment audio for detection in the audio stream.

It also allows for:
- offline keyword detection, where test audio is trimmed to contain only keywords of interest.
- online (streaming) keyword detection, where test audio has past and future context besides keywords of interest.### Languages

The MOCKS incorporates 5 languages:
- English - primary and largest test set,
- German,
- Spanish,
- French,
- Italian. passage: ## Dataset Structure

The MOCKS testset is split by language, source dataset, and OV-KWS type:


Each split is divided into:
- positive examples ('URL') - test examples with true keywords, 5000-8000 keywords in each subset,
- similar examples ('URL') - test examples with similar phrases to the keyword selected based on phonetic transcription distance,
- different examples ('URL') - test examples with completely different phrases.

All those files contain columns separated by tab:
- 'keyword_path' - path to audio containing keyword phrase.
- 'adversary_keyword_path' - path to test audio.
- 'adversary_keyword_timestamp_start' - start time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'adversary_keyword_timestamp_end' - end time in seconds of phrase of interest for a given keyword from 'keyword_path', the field only available in offline split.
- 'label' - whether the 'adversary_keyword_path' contain keyword from 'keyword_path' or not (1 - contains keyword, 0 - doesn't contain keyword).

Each split also contains a subset of whole data with the same field structure to allow faster evaluation ('URL.*.tsv').

Also, transcriptions are provided for each audio in:
- 'data_offline_transcription.tsv' - transcriptions for offline examples and 'keyword_path' from online scenario,
- 'data_online_transcription.tsv' - transcriptions for the adversary, test examples from online scenario,

three columns are present within each file:
- 'path_to_keyword'/'path_to_adversary_keyword' - path to the audio file,
- 'keyword_transcription'/'adversary_keyword_transcription' - audio transcription,
- 'keyword_phonetic_transcription'/'adversary_keyword_phonetic_transcription' - audio phonetic transcription.