
# The Vuk'uzenzele South African Multilingual Corpus

Github: [https://github.com/dsfsi/vukuzenzele-nlp/](https://github.com/dsfsi/vukuzenzele-nlp/)

Zenodo: [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7598539.svg)](https://doi.org/10.5281/zenodo.7598539)

Arxiv Preprint: [![arXiv](https://img.shields.io/badge/arXiv-2303.03750-b31b1b.svg)](https://arxiv.org/abs/2303.03750)

Give Feedback 📑: [DSFSI Resource Feedback Form](https://docs.google.com/forms/d/e/1FAIpQLSf7S36dyAUPx2egmXbFpnTBuzoRulhL5Elu-N1eoMhaO7v10w/formResponse)

# About
The dataset was obtained from the South African government magazine Vuk'uzenzele, created by the [Government Communication and Information System (GCIS)](https://www.gcis.gov.za/). 
The original raw PDFS were obtatined from the [Vuk'uzenzele website](https://www.vukuzenzele.gov.za/).

The datasets contain government magazine editions in 11 languages, namely:

|  Language  | Code  |  Language  | Code |
|------------|-------|------------|-------|
| English    | (eng) | Sepedi     | (sep) |
| Afrikaans  | (afr) | Setswana   | (tsn) |
| isiNdebele | (nbl) | Siswati    | (ssw) |
| isiXhosa   | (xho) | Tshivenda  | (ven) |
| isiZulu    | (zul) | Xitstonga  | (tso) |
| Sesotho    | (nso) |

## Available pairings

The alignment direction is bidrectional, i.e. xho-zul is zul-xho

afr-eng; afr-nbl; afr-nso; afr-sot; afr-ssw; afr-tsn; afr-tso; afr-ven; afr-xho; afr-zul  
eng-nbl; eng-nso; eng-sot ;eng-ssw; eng-tsn; eng-tso; eng-ven; eng-xho; eng-zul  
nbl-nso; nbl-sot; nbl-ssw; nbl-tsn; nbl-tso; nbl-ven; nbl-xho; nbl-zul  
nso-sot; nso-ssw; nso-tsn; nso-tso; nso-ven; nso-xho; nso-zul  
sot-ssw; sot-tsn; sot-tso; sot-ven; sot-xho; sot-zul  
ssw-tsn; ssw-tso; ssw-ven; ssw-xho; ssw-zul  
tsn-tso; tsn-ven; tsn-xho; tsn-zul  
tso-ven; tso-xho; tso-zul  
ven-xho; ven-zul  
xho-zul  

# Disclaimer

This dataset contains machine-readable data extracted from PDF documents, from https://www.vukuzenzele.gov.za/, provided by the Government Communication Information System (GCIS). While efforts were made to ensure the accuracy and completeness of this data, there may be errors or discrepancies between the original publications and this dataset. No warranties, guarantees or representations are given in relation to the information contained in the dataset. The members of the Data Science for Societal Impact Research Group bear no responsibility and/or liability for any such errors or discrepancies in this dataset. The Government Communication Information System (GCIS) bears no responsibility and/or liability for any such errors or discrepancies in this dataset. It is recommended that users verify all information contained herein before making decisions based upon this information.

# Datasets
The datasets consist of pairwise sentence aligned data. There are 55 distinct datasets of paired sentences.
The data is obtained by comparing [LASER](https://github.com/facebookresearch/LASER) embeddings of sentence tokens between 2 languages. If the similarity is high, the sentences are deemed semantic equivalents of one another and the observation is outputted. 

Naming convention:  
The naming structure of the pairwise_sentence_aligned folder is `aligned-{src_lang_code}-{tgt_lang_code}.csv`.  
For example, `aligned-afr-zul.csv` is the aligned sentences between Afrikaans and isiZulu. 

The data is in .csv format and the columns are `src_text`,`tgt_text`,`cosine_score` where:  
- `src_text` is the source sentence
- `tgt_text` is the target sentence
- `cosine_score` is the cosine similarity score obtained by comparing the sentence embeddings, it ranges from 0 to 1

**Note:** The notion of source (src) and target (tgt) are only necessary for distinction between the languages used in the aligned pair, as the sentence semantics should be bidirectional. (hallo <-> sawubona)

# Citation
Vukosi Marivate, Andani Madodonga, Daniel Njini, Richard Lastrucci, Isheanesu Dzingirai, Jenalea Rajab. **The Vuk'uzenzele South African Multilingual Corpus**, 2023

> @dataset{marivate_vukosi_2023_7598540,
  author       = {Marivate, Vukosi and
                  Njini, Daniel and
                  Madodonga, Andani and
                  Lastrucci, Richard and
                  Dzingirai, Isheanesu
                  Rajab, Jenalea},
  title        = {The Vuk'uzenzele South African Multilingual Corpus},
  month        = feb,
  year         = 2023,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.7598539},
  url          = {https://doi.org/10.5281/zenodo.7598539}
}

### Licence
* Licence for Data - [CC 4.0 BY](LICENSE.md)

task_categories:sentence-similarity task_categories:translation language:eng language:afr language:nbl language:xho language:zul language:sot language:nso language:tsn language:ssw language:ven language:tso license:cc-by-4.0 multilingual government arxiv:2303.03750 region:us
{"language": ["eng", "afr", "nbl", "xho", "zul", "sot", "nso", "tsn", "ssw", "ven", "tso"], "license": "cc-by-4.0", "task_categories": ["sentence-similarity", "translation"], "pretty_name": "The Vuk'uzenzele South African Multilingual Corpus", "tags": ["multilingual", "government"], "arxiv": 2303.0375, "configs": [{"config_name": "afr-eng", "data_files": [{"split": "train", "path": "afr-eng/train-*"}, {"split": "test", "path": "afr-eng/test-*"}, {"split": "eval", "path": "afr-eng/eval-*"}]}, {"config_name": "afr-nbl", "data_files": [{"split": "train", "path": "afr-nbl/train-*"}, {"split": "test", "path": "afr-nbl/test-*"}, {"split": "eval", "path": "afr-nbl/eval-*"}]}, {"config_name": "afr-nso", "data_files": [{"split": "train", "path": "afr-nso/train-*"}, {"split": "test", "path": "afr-nso/test-*"}, {"split": "eval", "path": "afr-nso/eval-*"}]}, {"config_name": "afr-sot", "data_files": [{"split": "train", "path": "afr-sot/train-*"}, {"split": "test", "path": "afr-sot/test-*"}, {"split": "eval", "path": "afr-sot/eval-*"}]}, {"config_name": "afr-ssw", "data_files": [{"split": "train", "path": "afr-ssw/train-*"}, {"split": "test", "path": "afr-ssw/test-*"}, {"split": "eval", "path": "afr-ssw/eval-*"}]}, {"config_name": "afr-tsn", "data_files": [{"split": "train", "path": "afr-tsn/train-*"}, {"split": "test", "path": "afr-tsn/test-*"}, {"split": "eval", "path": "afr-tsn/eval-*"}]}, {"config_name": "afr-tso", "data_files": [{"split": "train", "path": "afr-tso/train-*"}, {"split": "test", "path": "afr-tso/test-*"}, {"split": "eval", "path": "afr-tso/eval-*"}]}, {"config_name": "afr-ven", "data_files": [{"split": "train", "path": "afr-ven/train-*"}, {"split": "test", "path": "afr-ven/test-*"}, {"split": "eval", "path": "afr-ven/eval-*"}]}, {"config_name": "afr-xho", "data_files": [{"split": "train", "path": "afr-xho/train-*"}, {"split": "test", "path": "afr-xho/test-*"}, {"split": "eval", "path": "afr-xho/eval-*"}]}, {"config_name": "afr-zul", "data_files": [{"split": "train", "path": "afr-zul/train-*"}, {"split": "test", "path": "afr-zul/test-*"}, {"split": "eval", "path": "afr-zul/eval-*"}]}, {"config_name": "default", "data_files": [{"split": "train", "path": "data/train-*"}, {"split": "test", "path": "data/test-*"}]}, {"config_name": "eng-nbl", "data_files": [{"split": "train", "path": "eng-nbl/train-*"}, {"split": "test", "path": "eng-nbl/test-*"}, {"split": "eval", "path": "eng-nbl/eval-*"}]}, {"config_name": "eng-nso", "data_files": [{"split": "train", "path": "eng-nso/train-*"}, {"split": "test", "path": "eng-nso/test-*"}, {"split": "eval", "path": "eng-nso/eval-*"}]}, {"config_name": "eng-sot", "data_files": [{"split": "train", "path": "eng-sot/train-*"}, {"split": "test", "path": "eng-sot/test-*"}, {"split": "eval", "path": "eng-sot/eval-*"}]}, {"config_name": "eng-ssw", "data_files": [{"split": "train", "path": "eng-ssw/train-*"}, {"split": "test", "path": "eng-ssw/test-*"}, {"split": "eval", "path": "eng-ssw/eval-*"}]}, {"config_name": "eng-tsn", "data_files": [{"split": "train", "path": "eng-tsn/train-*"}, {"split": "test", "path": "eng-tsn/test-*"}, {"split": "eval", "path": "eng-tsn/eval-*"}]}, {"config_name": "eng-tso", "data_files": [{"split": "train", "path": "eng-tso/train-*"}, {"split": "test", "path": "eng-tso/test-*"}, {"split": "eval", "path": "eng-tso/eval-*"}]}, {"config_name": "eng-ven", "data_files": [{"split": "train", "path": "eng-ven/train-*"}, {"split": "test", "path": "eng-ven/test-*"}, {"split": "eval", "path": "eng-ven/eval-*"}]}, {"config_name": "eng-xho", "data_files": [{"split": "train", "path": "eng-xho/train-*"}, {"split": "test", "path": "eng-xho/test-*"}, {"split": "eval", "path": "eng-xho/eval-*"}]}, {"config_name": "eng-zul", "data_files": [{"split": "train", "path": "eng-zul/train-*"}, {"split": "test", "path": "eng-zul/test-*"}, {"split": "eval", "path": "eng-zul/eval-*"}]}, {"config_name": "nbl-nso", "data_files": [{"split": "train", "path": "nbl-nso/train-*"}, {"split": "test", "path": "nbl-nso/test-*"}, {"split": "eval", "path": "nbl-nso/eval-*"}]}, {"config_name": "nbl-sot", "data_files": [{"split": "train", "path": "nbl-sot/train-*"}, {"split": "test", "path": "nbl-sot/test-*"}, {"split": "eval", "path": "nbl-sot/eval-*"}]}, {"config_name": "nbl-ssw", "data_files": [{"split": "train", "path": "nbl-ssw/train-*"}, {"split": "test", "path": "nbl-ssw/test-*"}, {"split": "eval", "path": "nbl-ssw/eval-*"}]}, {"config_name": "nbl-tsn", "data_files": [{"split": "train", "path": "nbl-tsn/train-*"}, {"split": "test", "path": "nbl-tsn/test-*"}, {"split": "eval", "path": "nbl-tsn/eval-*"}]}, {"config_name": "nbl-tso", "data_files": [{"split": "train", "path": "nbl-tso/train-*"}, {"split": "test", "path": "nbl-tso/test-*"}, {"split": "eval", "path": "nbl-tso/eval-*"}]}, {"config_name": "nbl-ven", "data_files": [{"split": "train", "path": "nbl-ven/train-*"}, {"split": "test", "path": "nbl-ven/test-*"}, {"split": "eval", "path": "nbl-ven/eval-*"}]}, {"config_name": "nbl-xho", "data_files": [{"split": "train", "path": "nbl-xho/train-*"}, {"split": "test", "path": "nbl-xho/test-*"}, {"split": "eval", "path": "nbl-xho/eval-*"}]}, {"config_name": "nbl-zul", "data_files": [{"split": "train", "path": "nbl-zul/train-*"}, {"split": "test", "path": "nbl-zul/test-*"}, {"split": "eval", "path": "nbl-zul/eval-*"}]}, {"config_name": "nso-sot", "data_files": [{"split": "train", "path": "nso-sot/train-*"}, {"split": "test", "path": "nso-sot/test-*"}, {"split": "eval", "path": "nso-sot/eval-*"}]}, {"config_name": "nso-ssw", "data_files": [{"split": "train", "path": "nso-ssw/train-*"}, {"split": "test", "path": "nso-ssw/test-*"}, {"split": "eval", "path": "nso-ssw/eval-*"}]}, {"config_name": "nso-tsn", "data_files": [{"split": "train", "path": "nso-tsn/train-*"}, {"split": "test", "path": "nso-tsn/test-*"}, {"split": "eval", "path": "nso-tsn/eval-*"}]}, {"config_name": "nso-tso", "data_files": [{"split": "train", "path": "nso-tso/train-*"}, {"split": "test", "path": "nso-tso/test-*"}, {"split": "eval", "path": "nso-tso/eval-*"}]}, {"config_name": "nso-ven", "data_files": [{"split": "train", "path": "nso-ven/train-*"}, {"split": "test", "path": "nso-ven/test-*"}, {"split": "eval", "path": "nso-ven/eval-*"}]}, {"config_name": "nso-xho", "data_files": [{"split": "train", "path": "nso-xho/train-*"}, {"split": "test", "path": "nso-xho/test-*"}, {"split": "eval", "path": "nso-xho/eval-*"}]}, {"config_name": "nso-zul", "data_files": [{"split": "train", "path": "nso-zul/train-*"}, {"split": "test", "path": "nso-zul/test-*"}, {"split": "eval", "path": "nso-zul/eval-*"}]}, {"config_name": "sot-ssw", "data_files": [{"split": "train", "path": "sot-ssw/train-*"}, {"split": "test", "path": "sot-ssw/test-*"}, {"split": "eval", "path": "sot-ssw/eval-*"}]}, {"config_name": "sot-tsn", "data_files": [{"split": "train", "path": "sot-tsn/train-*"}, {"split": "test", "path": "sot-tsn/test-*"}, {"split": "eval", "path": "sot-tsn/eval-*"}]}, {"config_name": "sot-tso", "data_files": [{"split": "train", "path": "sot-tso/train-*"}, {"split": "test", "path": "sot-tso/test-*"}, {"split": "eval", "path": "sot-tso/eval-*"}]}, {"config_name": "sot-ven", "data_files": [{"split": "train", "path": "sot-ven/train-*"}, {"split": "test", "path": "sot-ven/test-*"}, {"split": "eval", "path": "sot-ven/eval-*"}]}, {"config_name": "sot-xho", "data_files": [{"split": "train", "path": "sot-xho/train-*"}, {"split": "test", "path": "sot-xho/test-*"}, {"split": "eval", "path": "sot-xho/eval-*"}]}, {"config_name": "sot-zul", "data_files": [{"split": "train", "path": "sot-zul/train-*"}, {"split": "test", "path": "sot-zul/test-*"}, {"split": "eval", "path": "sot-zul/eval-*"}]}, {"config_name": "ssw-tsn", "data_files": [{"split": "train", "path": "ssw-tsn/train-*"}, {"split": "test", "path": "ssw-tsn/test-*"}, {"split": "eval", "path": "ssw-tsn/eval-*"}]}, {"config_name": "ssw-tso", "data_files": [{"split": "train", "path": "ssw-tso/train-*"}, {"split": "test", "path": "ssw-tso/test-*"}, {"split": "eval", "path": "ssw-tso/eval-*"}]}, {"config_name": "ssw-ven", "data_files": [{"split": "train", "path": "ssw-ven/train-*"}, {"split": "test", "path": "ssw-ven/test-*"}, {"split": "eval", "path": "ssw-ven/eval-*"}]}, {"config_name": "ssw-xho", "data_files": [{"split": "train", "path": "ssw-xho/train-*"}, {"split": "test", "path": "ssw-xho/test-*"}, {"split": "eval", "path": "ssw-xho/eval-*"}]}, {"config_name": "ssw-zul", "data_files": [{"split": "train", "path": "ssw-zul/train-*"}, {"split": "test", "path": "ssw-zul/test-*"}, {"split": "eval", "path": "ssw-zul/eval-*"}]}, {"config_name": "tsn-tso", "data_files": [{"split": "train", "path": "tsn-tso/train-*"}, {"split": "test", "path": "tsn-tso/test-*"}, {"split": "eval", "path": "tsn-tso/eval-*"}]}, {"config_name": "tsn-ven", "data_files": [{"split": "train", "path": "tsn-ven/train-*"}, {"split": "test", "path": "tsn-ven/test-*"}, {"split": "eval", "path": "tsn-ven/eval-*"}]}, {"config_name": "tsn-xho", "data_files": [{"split": "train", "path": "tsn-xho/train-*"}, {"split": "test", "path": "tsn-xho/test-*"}, {"split": "eval", "path": "tsn-xho/eval-*"}]}, {"config_name": "tsn-zul", "data_files": [{"split": "train", "path": "tsn-zul/train-*"}, {"split": "test", "path": "tsn-zul/test-*"}, {"split": "eval", "path": "tsn-zul/eval-*"}]}, {"config_name": "tso-ven", "data_files": [{"split": "train", "path": "tso-ven/train-*"}, {"split": "test", "path": "tso-ven/test-*"}, {"split": "eval", "path": "tso-ven/eval-*"}]}, {"config_name": "tso-xho", "data_files": [{"split": "train", "path": "tso-xho/train-*"}, {"split": "test", "path": "tso-xho/test-*"}, {"split": "eval", "path": "tso-xho/eval-*"}]}, {"config_name": "tso-zul", "data_files": [{"split": "train", "path": "tso-zul/train-*"}, {"split": "test", "path": "tso-zul/test-*"}, {"split": "eval", "path": "tso-zul/eval-*"}]}, {"config_name": "ven-xho", "data_files": [{"split": "train", "path": "ven-xho/train-*"}, {"split": "test", "path": "ven-xho/test-*"}, {"split": "eval", "path": "ven-xho/eval-*"}]}, {"config_name": "ven-zul", "data_files": [{"split": "train", "path": "ven-zul/train-*"}, {"split": "test", "path": "ven-zul/test-*"}, {"split": "eval", "path": "ven-zul/eval-*"}]}, {"config_name": "xho-zul", "data_files": [{"split": "train", "path": "xho-zul/train-*"}, {"split": "test", "path": "xho-zul/test-*"}, {"split": "eval", "path": "xho-zul/eval-*"}]}], "dataset_info": [{"config_name": "afr-eng", "features": [{"name": "afr", "dtype": "string"}, {"name": "eng", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 793530, "num_examples": 2660}, {"name": "test", "num_bytes": 171644, "num_examples": 570}, {"name": "eval", "num_bytes": 172132, "num_examples": 571}], "download_size": 757198, "dataset_size": 1137306}, {"config_name": "afr-nbl", "features": [{"name": "afr", "dtype": "string"}, {"name": "nbl", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 281328, "num_examples": 723}, {"name": "test", "num_bytes": 57947, "num_examples": 155}, {"name": "eval", "num_bytes": 59996, "num_examples": 155}], "download_size": 279950, "dataset_size": 399271}, {"config_name": "afr-nso", "features": [{"name": "afr", "dtype": "string"}, {"name": "nso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 980475, "num_examples": 2937}, {"name": "test", "num_bytes": 203451, "num_examples": 630}, {"name": "eval", "num_bytes": 214623, "num_examples": 630}], "download_size": 892392, "dataset_size": 1398549}, {"config_name": "afr-sot", "features": [{"name": "afr", "dtype": "string"}, {"name": "sot", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 971841, "num_examples": 2968}, {"name": "test", "num_bytes": 211374, "num_examples": 636}, {"name": "eval", "num_bytes": 209697, "num_examples": 636}], "download_size": 901006, "dataset_size": 1392912}, {"config_name": "afr-ssw", "features": [{"name": "afr", "dtype": "string"}, {"name": "ssw", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 975744, "num_examples": 3042}, {"name": "test", "num_bytes": 209151, "num_examples": 652}, {"name": "eval", "num_bytes": 208877, "num_examples": 653}], "download_size": 927666, "dataset_size": 1393772}, {"config_name": "afr-tsn", "features": [{"name": "afr", "dtype": "string"}, {"name": "tsn", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1010506, "num_examples": 2830}, {"name": "test", "num_bytes": 218153, "num_examples": 607}, {"name": "eval", "num_bytes": 214373, "num_examples": 607}], "download_size": 913596, "dataset_size": 1443032}, {"config_name": "afr-tso", "features": [{"name": "afr", "dtype": "string"}, {"name": "tso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 980025, "num_examples": 2952}, {"name": "test", "num_bytes": 213355, "num_examples": 633}, {"name": "eval", "num_bytes": 211642, "num_examples": 633}], "download_size": 902666, "dataset_size": 1405022}, {"config_name": "afr-ven", "features": [{"name": "afr", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 352031, "num_examples": 830}, {"name": "test", "num_bytes": 72702, "num_examples": 178}, {"name": "eval", "num_bytes": 75243, "num_examples": 178}], "download_size": 323825, "dataset_size": 499976}, {"config_name": "afr-xho", "features": [{"name": "afr", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 986062, "num_examples": 3098}, {"name": "test", "num_bytes": 205229, "num_examples": 664}, {"name": "eval", "num_bytes": 210379, "num_examples": 665}], "download_size": 944334, "dataset_size": 1401670}, {"config_name": "afr-zul", "features": [{"name": "afr", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 982102, "num_examples": 3078}, {"name": "test", "num_bytes": 208473, "num_examples": 660}, {"name": "eval", "num_bytes": 201824, "num_examples": 660}], "download_size": 932565, "dataset_size": 1392399}, {"config_name": "default", "features": [{"name": "nbl", "dtype": "string"}, {"name": "nso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 128131, "num_examples": 315}, {"name": "test", "num_bytes": 31826, "num_examples": 79}], "download_size": 113394, "dataset_size": 159957}, {"config_name": "eng-nbl", "features": [{"name": "eng", "dtype": "string"}, {"name": "nbl", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 78419, "num_examples": 207}, {"name": "test", "num_bytes": 16930, "num_examples": 45}, {"name": "eval", "num_bytes": 15202, "num_examples": 45}], "download_size": 89654, "dataset_size": 110551}, {"config_name": "eng-nso", "features": [{"name": "eng", "dtype": "string"}, {"name": "nso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 848347, "num_examples": 2616}, {"name": "test", "num_bytes": 183267, "num_examples": 561}, {"name": "eval", "num_bytes": 181802, "num_examples": 561}], "download_size": 770909, "dataset_size": 1213416}, {"config_name": "eng-sot", "features": [{"name": "eng", "dtype": "string"}, {"name": "sot", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 843173, "num_examples": 2671}, {"name": "test", "num_bytes": 181709, "num_examples": 573}, {"name": "eval", "num_bytes": 180602, "num_examples": 573}], "download_size": 776145, "dataset_size": 1205484}, {"config_name": "eng-ssw", "features": [{"name": "eng", "dtype": "string"}, {"name": "ssw", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 830354, "num_examples": 2662}, {"name": "test", "num_bytes": 175688, "num_examples": 571}, {"name": "eval", "num_bytes": 176734, "num_examples": 571}], "download_size": 777951, "dataset_size": 1182776}, {"config_name": "eng-tsn", "features": [{"name": "eng", "dtype": "string"}, {"name": "tsn", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 880306, "num_examples": 2517}, {"name": "test", "num_bytes": 190843, "num_examples": 539}, {"name": "eval", "num_bytes": 187728, "num_examples": 540}], "download_size": 786563, "dataset_size": 1258877}, {"config_name": "eng-tso", "features": [{"name": "eng", "dtype": "string"}, {"name": "tso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 849914, "num_examples": 2623}, {"name": "test", "num_bytes": 181181, "num_examples": 562}, {"name": "eval", "num_bytes": 176362, "num_examples": 563}], "download_size": 773662, "dataset_size": 1207457}, {"config_name": "eng-ven", "features": [{"name": "eng", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 113647, "num_examples": 279}, {"name": "test", "num_bytes": 26195, "num_examples": 60}, {"name": "eval", "num_bytes": 26121, "num_examples": 60}], "download_size": 119271, "dataset_size": 165963}, {"config_name": "eng-xho", "features": [{"name": "eng", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 827201, "num_examples": 2662}, {"name": "test", "num_bytes": 175023, "num_examples": 571}, {"name": "eval", "num_bytes": 176047, "num_examples": 571}], "download_size": 784961, "dataset_size": 1178271}, {"config_name": "eng-zul", "features": [{"name": "eng", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 815837, "num_examples": 2646}, {"name": "test", "num_bytes": 168769, "num_examples": 567}, {"name": "eval", "num_bytes": 177547, "num_examples": 567}], "download_size": 767836, "dataset_size": 1162153}, {"config_name": "nbl-nso", "features": [{"name": "nbl", "dtype": "string"}, {"name": "nso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 109438, "num_examples": 275}, {"name": "test", "num_bytes": 24000, "num_examples": 59}, {"name": "eval", "num_bytes": 26519, "num_examples": 60}], "download_size": 118816, "dataset_size": 159957}, {"config_name": "nbl-sot", "features": [{"name": "nbl", "dtype": "string"}, {"name": "sot", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 149593, "num_examples": 365}, {"name": "test", "num_bytes": 30656, "num_examples": 78}, {"name": "eval", "num_bytes": 32211, "num_examples": 79}], "download_size": 152576, "dataset_size": 212460}, {"config_name": "nbl-ssw", "features": [{"name": "nbl", "dtype": "string"}, {"name": "ssw", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 146544, "num_examples": 387}, {"name": "test", "num_bytes": 33410, "num_examples": 83}, {"name": "eval", "num_bytes": 32858, "num_examples": 84}], "download_size": 157314, "dataset_size": 212812}, {"config_name": "nbl-tsn", "features": [{"name": "nbl", "dtype": "string"}, {"name": "tsn", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 131856, "num_examples": 302}, {"name": "test", "num_bytes": 31961, "num_examples": 65}, {"name": "eval", "num_bytes": 29676, "num_examples": 65}], "download_size": 139315, "dataset_size": 193493}, {"config_name": "nbl-tso", "features": [{"name": "nbl", "dtype": "string"}, {"name": "tso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 127433, "num_examples": 296}, {"name": "test", "num_bytes": 24654, "num_examples": 63}, {"name": "eval", "num_bytes": 23290, "num_examples": 64}], "download_size": 127532, "dataset_size": 175377}, {"config_name": "nbl-ven", "features": [{"name": "nbl", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 976911, "num_examples": 2660}, {"name": "test", "num_bytes": 211536, "num_examples": 570}, {"name": "eval", "num_bytes": 207694, "num_examples": 570}], "download_size": 885066, "dataset_size": 1396141}, {"config_name": "nbl-xho", "features": [{"name": "nbl", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 149020, "num_examples": 403}, {"name": "test", "num_bytes": 33319, "num_examples": 87}, {"name": "eval", "num_bytes": 31809, "num_examples": 87}], "download_size": 160427, "dataset_size": 214148}, {"config_name": "nbl-zul", "features": [{"name": "nbl", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 111905, "num_examples": 289}, {"name": "test", "num_bytes": 25799, "num_examples": 62}, {"name": "eval", "num_bytes": 22660, "num_examples": 63}], "download_size": 124588, "dataset_size": 160364}, {"config_name": "nso-sot", "features": [{"name": "nso", "dtype": "string"}, {"name": "sot", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1057851, "num_examples": 3052}, {"name": "test", "num_bytes": 226420, "num_examples": 654}, {"name": "eval", "num_bytes": 232934, "num_examples": 655}], "download_size": 945243, "dataset_size": 1517205}, {"config_name": "nso-ssw", "features": [{"name": "nso", "dtype": "string"}, {"name": "ssw", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1020738, "num_examples": 2968}, {"name": "test", "num_bytes": 219932, "num_examples": 636}, {"name": "eval", "num_bytes": 218161, "num_examples": 637}], "download_size": 922981, "dataset_size": 1458831}, {"config_name": "nso-tsn", "features": [{"name": "nso", "dtype": "string"}, {"name": "tsn", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1092885, "num_examples": 2918}, {"name": "test", "num_bytes": 238439, "num_examples": 625}, {"name": "eval", "num_bytes": 234644, "num_examples": 626}], "download_size": 952272, "dataset_size": 1565968}, {"config_name": "nso-tso", "features": [{"name": "nso", "dtype": "string"}, {"name": "tso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1045959, "num_examples": 2956}, {"name": "test", "num_bytes": 228677, "num_examples": 634}, {"name": "eval", "num_bytes": 226626, "num_examples": 634}], "download_size": 925262, "dataset_size": 1501262}, {"config_name": "nso-ven", "features": [{"name": "nso", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 155499, "num_examples": 343}, {"name": "test", "num_bytes": 35576, "num_examples": 73}, {"name": "eval", "num_bytes": 31381, "num_examples": 74}], "download_size": 152424, "dataset_size": 222456}, {"config_name": "nso-xho", "features": [{"name": "nso", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1022670, "num_examples": 2959}, {"name": "test", "num_bytes": 214850, "num_examples": 634}, {"name": "eval", "num_bytes": 212932, "num_examples": 635}], "download_size": 929486, "dataset_size": 1450452}, {"config_name": "nso-zul", "features": [{"name": "nso", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1014985, "num_examples": 2998}, {"name": "test", "num_bytes": 223825, "num_examples": 643}, {"name": "eval", "num_bytes": 219173, "num_examples": 643}], "download_size": 926742, "dataset_size": 1457983}, {"config_name": "sot-ssw", "features": [{"name": "sot", "dtype": "string"}, {"name": "ssw", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1036642, "num_examples": 3075}, {"name": "test", "num_bytes": 217328, "num_examples": 659}, {"name": "eval", "num_bytes": 222863, "num_examples": 660}], "download_size": 950426, "dataset_size": 1476833}, {"config_name": "sot-tsn", "features": [{"name": "sot", "dtype": "string"}, {"name": "tsn", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1118346, "num_examples": 3019}, {"name": "test", "num_bytes": 237826, "num_examples": 647}, {"name": "eval", "num_bytes": 235279, "num_examples": 647}], "download_size": 981019, "dataset_size": 1591451}, {"config_name": "sot-tso", "features": [{"name": "sot", "dtype": "string"}, {"name": "tso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1057631, "num_examples": 3027}, {"name": "test", "num_bytes": 226229, "num_examples": 649}, {"name": "eval", "num_bytes": 222671, "num_examples": 649}], "download_size": 943068, "dataset_size": 1506531}, {"config_name": "sot-ven", "features": [{"name": "sot", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 201166, "num_examples": 461}, {"name": "test", "num_bytes": 44845, "num_examples": 99}, {"name": "eval", "num_bytes": 42607, "num_examples": 99}], "download_size": 191947, "dataset_size": 288618}, {"config_name": "sot-xho", "features": [{"name": "sot", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1042930, "num_examples": 3098}, {"name": "test", "num_bytes": 217327, "num_examples": 664}, {"name": "eval", "num_bytes": 223313, "num_examples": 665}], "download_size": 964792, "dataset_size": 1483570}, {"config_name": "sot-zul", "features": [{"name": "sot", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1017097, "num_examples": 3079}, {"name": "test", "num_bytes": 223761, "num_examples": 660}, {"name": "eval", "num_bytes": 227514, "num_examples": 660}], "download_size": 949761, "dataset_size": 1468372}, {"config_name": "ssw-tsn", "features": [{"name": "ssw", "dtype": "string"}, {"name": "tsn", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1124400, "num_examples": 3110}, {"name": "test", "num_bytes": 238160, "num_examples": 666}, {"name": "eval", "num_bytes": 246176, "num_examples": 667}], "download_size": 1012570, "dataset_size": 1608736}, {"config_name": "ssw-tso", "features": [{"name": "ssw", "dtype": "string"}, {"name": "tso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1069540, "num_examples": 3142}, {"name": "test", "num_bytes": 237608, "num_examples": 673}, {"name": "eval", "num_bytes": 231657, "num_examples": 674}], "download_size": 980833, "dataset_size": 1538805}, {"config_name": "ssw-ven", "features": [{"name": "ssw", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 170825, "num_examples": 401}, {"name": "test", "num_bytes": 34774, "num_examples": 86}, {"name": "eval", "num_bytes": 39434, "num_examples": 87}], "download_size": 170522, "dataset_size": 245033}, {"config_name": "ssw-xho", "features": [{"name": "ssw", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1047037, "num_examples": 3193}, {"name": "test", "num_bytes": 227505, "num_examples": 684}, {"name": "eval", "num_bytes": 219981, "num_examples": 685}], "download_size": 992683, "dataset_size": 1494523}, {"config_name": "ssw-zul", "features": [{"name": "ssw", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1054772, "num_examples": 3255}, {"name": "test", "num_bytes": 231524, "num_examples": 698}, {"name": "eval", "num_bytes": 223701, "num_examples": 698}], "download_size": 997182, "dataset_size": 1509997}, {"config_name": "tsn-tso", "features": [{"name": "tsn", "dtype": "string"}, {"name": "tso", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1141163, "num_examples": 3023}, {"name": "test", "num_bytes": 244100, "num_examples": 648}, {"name": "eval", "num_bytes": 242886, "num_examples": 648}], "download_size": 998631, "dataset_size": 1628149}, {"config_name": "tsn-ven", "features": [{"name": "tsn", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 175353, "num_examples": 361}, {"name": "test", "num_bytes": 39141, "num_examples": 77}, {"name": "eval", "num_bytes": 37453, "num_examples": 78}], "download_size": 165408, "dataset_size": 251947}, {"config_name": "tsn-xho", "features": [{"name": "tsn", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1095514, "num_examples": 2989}, {"name": "test", "num_bytes": 231544, "num_examples": 640}, {"name": "eval", "num_bytes": 227856, "num_examples": 641}], "download_size": 986295, "dataset_size": 1554914}, {"config_name": "tsn-zul", "features": [{"name": "tsn", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1090114, "num_examples": 3038}, {"name": "test", "num_bytes": 232488, "num_examples": 651}, {"name": "eval", "num_bytes": 240758, "num_examples": 651}], "download_size": 989654, "dataset_size": 1563360}, {"config_name": "tso-ven", "features": [{"name": "tso", "dtype": "string"}, {"name": "ven", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 171428, "num_examples": 375}, {"name": "test", "num_bytes": 33029, "num_examples": 80}, {"name": "eval", "num_bytes": 38079, "num_examples": 81}], "download_size": 163896, "dataset_size": 242536}, {"config_name": "tso-xho", "features": [{"name": "tso", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1049926, "num_examples": 3066}, {"name": "test", "num_bytes": 224708, "num_examples": 657}, {"name": "eval", "num_bytes": 221699, "num_examples": 657}], "download_size": 967978, "dataset_size": 1496333}, {"config_name": "tso-zul", "features": [{"name": "tso", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1059314, "num_examples": 3106}, {"name": "test", "num_bytes": 224935, "num_examples": 666}, {"name": "eval", "num_bytes": 225248, "num_examples": 666}], "download_size": 970505, "dataset_size": 1509497}, {"config_name": "ven-xho", "features": [{"name": "ven", "dtype": "string"}, {"name": "xho", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 164305, "num_examples": 401}, {"name": "test", "num_bytes": 36290, "num_examples": 86}, {"name": "eval", "num_bytes": 35520, "num_examples": 87}], "download_size": 165177, "dataset_size": 236115}, {"config_name": "ven-zul", "features": [{"name": "ven", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 138178, "num_examples": 336}, {"name": "test", "num_bytes": 32949, "num_examples": 72}, {"name": "eval", "num_bytes": 30697, "num_examples": 72}], "download_size": 143542, "dataset_size": 201824}, {"config_name": "xho-zul", "features": [{"name": "xho", "dtype": "string"}, {"name": "zul", "dtype": "string"}, {"name": "score", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1062980, "num_examples": 3276}, {"name": "test", "num_bytes": 226001, "num_examples": 702}, {"name": "eval", "num_bytes": 225893, "num_examples": 703}], "download_size": 1011124, "dataset_size": 1514874}]}
TAGS
#task_categories-sentence-similarity #task_categories-translation #language-English #language-Afrikaans #language-South Ndebele #language-Xhosa #language-Zulu #language-Southern Sotho #language-Pedi #language-Tswana #language-Swati #language-Venda #language-Tsonga #license-cc-by-4.0 #multilingual #government #arxiv-2303.03750 #region-us 

The Vuk'uzenzele South African Multilingual Corpus
==================================================


Github: URL


Zenodo: ![DOI](URL


Arxiv Preprint: ![arXiv](URL


Give Feedback : DSFSI Resource Feedback Form


About
=====


The dataset was obtained from the South African government magazine Vuk'uzenzele, created by the Government Communication and Information System (GCIS).
The original raw PDFS were obtatined from the Vuk'uzenzele website.


The datasets contain government magazine editions in 11 languages, namely:



Available pairings
------------------


The alignment direction is bidrectional, i.e. xho-zul is zul-xho


afr-eng; afr-nbl; afr-nso; afr-sot; afr-ssw; afr-tsn; afr-tso; afr-ven; afr-xho; afr-zul  

eng-nbl; eng-nso; eng-sot ;eng-ssw; eng-tsn; eng-tso; eng-ven; eng-xho; eng-zul  

nbl-nso; nbl-sot; nbl-ssw; nbl-tsn; nbl-tso; nbl-ven; nbl-xho; nbl-zul  

nso-sot; nso-ssw; nso-tsn; nso-tso; nso-ven; nso-xho; nso-zul  

sot-ssw; sot-tsn; sot-tso; sot-ven; sot-xho; sot-zul  

ssw-tsn; ssw-tso; ssw-ven; ssw-xho; ssw-zul  

tsn-tso; tsn-ven; tsn-xho; tsn-zul  

tso-ven; tso-xho; tso-zul  

ven-xho; ven-zul  

xho-zul


Disclaimer
==========


This dataset contains machine-readable data extracted from PDF documents, from URL provided by the Government Communication Information System (GCIS). While efforts were made to ensure the accuracy and completeness of this data, there may be errors or discrepancies between the original publications and this dataset. No warranties, guarantees or representations are given in relation to the information contained in the dataset. The members of the Data Science for Societal Impact Research Group bear no responsibility and/or liability for any such errors or discrepancies in this dataset. The Government Communication Information System (GCIS) bears no responsibility and/or liability for any such errors or discrepancies in this dataset. It is recommended that users verify all information contained herein before making decisions based upon this information.


Datasets
========


The datasets consist of pairwise sentence aligned data. There are 55 distinct datasets of paired sentences.
The data is obtained by comparing LASER embeddings of sentence tokens between 2 languages. If the similarity is high, the sentences are deemed semantic equivalents of one another and the observation is outputted.


Naming convention:  

The naming structure of the pairwise\_sentence\_aligned folder is 'aligned-{src\_lang\_code}-{tgt\_lang\_code}.csv'.  

For example, 'URL' is the aligned sentences between Afrikaans and isiZulu.


The data is in .csv format and the columns are 'src\_text','tgt\_text','cosine\_score' where:


* 'src\_text' is the source sentence
* 'tgt\_text' is the target sentence
* 'cosine\_score' is the cosine similarity score obtained by comparing the sentence embeddings, it ranges from 0 to 1


Note: The notion of source (src) and target (tgt) are only necessary for distinction between the languages used in the aligned pair, as the sentence semantics should be bidirectional. (hallo <-> sawubona)


Vukosi Marivate, Andani Madodonga, Daniel Njini, Richard Lastrucci, Isheanesu Dzingirai, Jenalea Rajab. The Vuk'uzenzele South African Multilingual Corpus, 2023



> 
> @dataset{marivate\_vukosi\_2023\_7598540,
> author = {Marivate, Vukosi and
> Njini, Daniel and
> Madodonga, Andani and
> Lastrucci, Richard and
> Dzingirai, Isheanesu
> Rajab, Jenalea},
> title = {The Vuk'uzenzele South African Multilingual Corpus},
> month = feb,
> year = 2023,
> publisher = {Zenodo},
> doi = {10.5281/zenodo.7598539},
> url = {URL
> }
> 
> 
> 


### Licence


* Licence for Data - CC 4.0 BY



### Licence


* Licence for Data - CC 4.0 BY
TAGS
#task_categories-sentence-similarity #task_categories-translation #language-English #language-Afrikaans #language-South Ndebele #language-Xhosa #language-Zulu #language-Southern Sotho #language-Pedi #language-Tswana #language-Swati #language-Venda #language-Tsonga #license-cc-by-4.0 #multilingual #government #arxiv-2303.03750 #region-us 
 ### Licence


* Licence for Data - CC 4.0 BY
passage: TAGS
#task_categories-sentence-similarity #task_categories-translation #language-English #language-Afrikaans #language-South Ndebele #language-Xhosa #language-Zulu #language-Southern Sotho #language-Pedi #language-Tswana #language-Swati #language-Venda #language-Tsonga #license-cc-by-4.0 #multilingual #government #arxiv-2303.03750 #region-us 
### Licence


* Licence for Data - CC 4.0 BY