# Machine translation dataset for NLU (Virual Assistant) with slot transfer between languages
version: 0.5.1
## Dataset Summary

Disclaimer: This is for research purposes only. Please have a look at the license section below. Some of the datasets used to construct IVA_MT have an unknown license.

IVA_MT is a machine translation dataset that can be used to train, adapt and evaluate MT models used in Virtual Assistant NLU context (e.g. to translate trainig corpus of NLU).

## Dataset Composition

### en-pl

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 11514  | 2033  | 2974  |
| [Leyzer 0.2.0](https://github.com/cartesinus/leyzer/tree/0.2.0)      | 3974   | 701   | 1380  |
| [OpenSubtitles from OPUS](https://opus.nlpl.eu/OpenSubtitles-v1.php) | 2329   | 411   | 500   |
| [KDE from OPUS](https://opus.nlpl.eu/KDE4.php)                       | 1154   | 241   | 241   |
| [CCMatrix from Opus](https://opus.nlpl.eu/CCMatrix.php)              | 1096   | 232   | 237   |
| [Ubuntu from OPUS](https://opus.nlpl.eu/Ubuntu.php)                  | 281    | 60    | 59    |
| [Gnome from OPUS](https://opus.nlpl.eu/GNOME.php)                    | 14     | 3     | 3     |
| *total*                                                              | 20362  | 3681  | 5394  |

### en-de

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 7536   | 1346  | 1955  |

### en-es

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 8415   | 1526  | 2202  |

### en-sv

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 7540   | 1360  | 1921  |

### en-fr

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 6800   | 1203  | 1757  |

### en-pt

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 7368   | 1296  | 1885  |

### en-hi

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 6702   | 1175  | 1747  |

### en-tr

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 8269   | 1474  | 2170  |

### en-ja

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 8066   | 1434  | 2085  |

### en-zh

| Corpus                                                               | Train  | Dev   | Test  |
|----------------------------------------------------------------------|--------|-------|-------|
| [Massive 1.1](https://huggingface.co/datasets/AmazonScience/massive) | 8433   | 1513  | 2179  |
| ChatGPT                                                              | 1312   | 200   | 200   |


## Tools
Scripts used to generate this dataset can be found on [github](https://github.com/cartesinus/iva_mt).

## Citation

If you use this models please cite:
```
@article{Sowanski2023SlotLI,
  title={Slot Lost in Translation? Not Anymore: A Machine Translation Model for Virtual Assistants with Type-Independent Slot Transfer},
  author={Marcin Sowanski and Artur Janicki},
  journal={2023 30th International Conference on Systems, Signals and Image Processing (IWSSIP)},
  year={2023},
  pages={1-5}
}
```

## License
This is a composition of 7 datasets, and the license is as defined in original release:
- MASSIVE: [CC-BY 4.0](https://huggingface.co/datasets/AmazonScience/massive/blob/main/LICENSE)
- Leyzer: [CC BY-NC 4.0](https://github.com/cartesinus/leyzer/blob/master/LICENSE)
- OpenSubtitles: unknown
- KDE: [GNU Public License](https://l10n.kde.org/about.php)
- CCMatrix: no license given, therefore assuming it is LASER project license [BSD](https://github.com/facebookresearch/LASER/blob/main/LICENSE)
- Ubuntu: [GNU Public License](https://help.launchpad.net/Legal)
- Gnome: unknown

task_categories:translation size_categories:10K<n<100K language:en language:pl language:de language:es language:sv language:fr language:pt license:cc-by-4.0 machine translation nlu natural-language-understanding virtual assistant region:us
{"language": ["en", "pl", "de", "es", "sv", "fr", "pt"], "license": "cc-by-4.0", "size_categories": ["10K<n<100K"], "task_categories": ["translation"], "pretty_name": "Machine translation for NLU with slot transfer", "dataset_info": {"features": [{"name": "id", "dtype": "string"}, {"name": "locale", "dtype": "string"}, {"name": "origin", "dtype": "string"}, {"name": "partition", "dtype": "string"}, {"name": "translation_utt", "dtype": {"translation": {"languages": ["en", "pl"]}}}, {"name": "translation_xml", "dtype": {"translation": {"languages": ["en", "pl"]}}}, {"name": "src_bio", "dtype": "string"}, {"name": "tgt_bio", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6187206, "num_examples": 20362}, {"name": "validation", "num_bytes": 1115480, "num_examples": 3681}, {"name": "test", "num_bytes": 1587613, "num_examples": 5394}], "download_size": 3851892, "dataset_size": 8890299}, "tags": ["machine translation", "nlu", "natural-language-understanding", "virtual assistant"]}
TAGS
#task_categories-translation #size_categories-10K<n<100K #language-English #language-Polish #language-German #language-Spanish #language-Swedish #language-French #language-Portuguese #license-cc-by-4.0 #machine translation #nlu #natural-language-understanding #virtual assistant #region-us 

Machine translation dataset for NLU (Virual Assistant) with slot transfer between languages
===========================================================================================


version: 0.5.1


Dataset Summary
---------------


Disclaimer: This is for research purposes only. Please have a look at the license section below. Some of the datasets used to construct IVA\_MT have an unknown license.


IVA\_MT is a machine translation dataset that can be used to train, adapt and evaluate MT models used in Virtual Assistant NLU context (e.g. to translate trainig corpus of NLU).


Dataset Composition
-------------------


### en-pl



### en-de



### en-es



### en-sv



### en-fr



### en-pt



### en-hi



### en-tr



### en-ja



### en-zh



Tools
-----


Scripts used to generate this dataset can be found on github.


If you use this models please cite:


License
-------


This is a composition of 7 datasets, and the license is as defined in original release:


* MASSIVE: CC-BY 4.0
* Leyzer: CC BY-NC 4.0
* OpenSubtitles: unknown
* KDE: GNU Public License
* CCMatrix: no license given, therefore assuming it is LASER project license BSD
* Ubuntu: GNU Public License
* Gnome: unknown



### en-pl ### en-de ### en-es ### en-sv ### en-fr ### en-pt ### en-hi ### en-tr ### en-ja ### en-zh



Tools
-----


Scripts used to generate this dataset can be found on github.


If you use this models please cite:


License
-------


This is a composition of 7 datasets, and the license is as defined in original release:


* MASSIVE: CC-BY 4.0
* Leyzer: CC BY-NC 4.0
* OpenSubtitles: unknown
* KDE: GNU Public License
* CCMatrix: no license given, therefore assuming it is LASER project license BSD
* Ubuntu: GNU Public License
* Gnome: unknown
TAGS
#task_categories-translation #size_categories-10K<n<100K #language-English #language-Polish #language-German #language-Spanish #language-Swedish #language-French #language-Portuguese #license-cc-by-4.0 #machine translation #nlu #natural-language-understanding #virtual assistant #region-us 
 ### en-pl ### en-de ### en-es ### en-sv ### en-fr ### en-pt ### en-hi ### en-tr ### en-ja ### en-zh



Tools
-----


Scripts used to generate this dataset can be found on github.


If you use this models please cite:


License
-------


This is a composition of 7 datasets, and the license is as defined in original release:


* MASSIVE: CC-BY 4.0
* Leyzer: CC BY-NC 4.0
* OpenSubtitles: unknown
* KDE: GNU Public License
* CCMatrix: no license given, therefore assuming it is LASER project license BSD
* Ubuntu: GNU Public License
* Gnome: unknown
passage: TAGS
#task_categories-translation #size_categories-10K<n<100K #language-English #language-Polish #language-German #language-Spanish #language-Swedish #language-French #language-Portuguese #license-cc-by-4.0 #machine translation #nlu #natural-language-understanding #virtual assistant #region-us 
### en-pl### en-de### en-es### en-sv### en-fr### en-pt### en-hi### en-tr### en-ja### en-zh



Tools
-----


Scripts used to generate this dataset can be found on github.


If you use this models please cite:


License
-------


This is a composition of 7 datasets, and the license is as defined in original release:


* MASSIVE: CC-BY 4.0
* Leyzer: CC BY-NC 4.0
* OpenSubtitles: unknown
* KDE: GNU Public License
* CCMatrix: no license given, therefore assuming it is LASER project license BSD
* Ubuntu: GNU Public License
* Gnome: unknown