
# Dataset Card for cantonese-mandarin-translations

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:**
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

This is a machine-translated parallel corpus between Cantonese (a Chinese dialect that is mainly spoken by Guangdong (province of China), Hong Kong, Macau and part of Malaysia) and Chinese (written form, in Simplified Chinese).
 

### Supported Tasks and Leaderboards

N/A

### Languages

- Cantonese (`yue`)
- Simplified Chinese (`zh-CN`)

## Dataset Structure

JSON lines with `yue` field and `zh` field for the parallel corpus.

### Data Instances

N/A

### Data Fields

- `yue`: Cantonese corpus
- `zh`: translated Chinese corpus

### Data Splits

No data splitting is done as of yet.

## Dataset Creation

The dataset is produced by doing the following:

- Download [HKCancor Cantonese Corpus](https://github.com/fcbond/hkcancor) and [CommonVoice Cantonese (Hong Kong Chinese `yue`) text corpus](https://commonvoice.mozilla.org/en/datasets)
- Extract text corpus and merge datasets
- Run text against [Microsoft's Translator API](https://learn.microsoft.com/en-us/azure/ai-services/translator/language-support) from `yue` to `zh-Hans`

### Curation Rationale

Currently no such corpus exists, and it is hard to find such a corpus, so we tried to generate a reasonable batch of samples using machine translation for research purposes.

### Source Data

- [HKCancor](https://github.com/fcbond/hkcancor)
- [CommonVoice 7.0 Chinese (Hong Kong)](https://commonvoice.mozilla.org/en/datasets)

#### Initial Data Collection and Normalization

Normalization scripts will be included soon.

#### Who are the source language producers?

- [HKCancor](https://github.com/fcbond/hkcancor)
- [CommonVoice 7.0 Chinese (Hong Kong)](https://commonvoice.mozilla.org/en/datasets)

### Annotations

#### Annotation process

We run the Cantonese text corpus against Microsoft's Translator API.

#### Who are the annotators?

- [Microsoft's Translator API](https://learn.microsoft.com/en-us/azure/ai-services/translator/language-support)

### Personal and Sensitive Information

N/A

## Considerations for Using the Data

### Social Impact of Dataset

We would like to share this parallel corpus and welcome contributions to preserve the Cantonese dialect.

### Discussion of Biases

N/A

### Other Known Limitations

This parallel corpus is machine-translated, it is not 100% accurate.

## Additional Information

### Dataset Curators

- [Botisan AI](https://botisan.ai)
- [Haoran (Simon) Liang](https://github.com/lhr0909)

### Licensing Information

[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)

### Citation Information

```
@misc {botisanAiCantoneseMandarinTranslationsDatasets,
    author = {Liang, H.},
    title = {Cantonese Mandarin Translations Dataset},
    year = {2021},
    url = {https://huggingface.co/datasets/botisan-ai/cantonese-mandarin-translations},
}
```

### Contributions

Thanks to [@lhr0909](https://github.com/lhr0909) for adding this dataset.
task_categories:text2text-generation task_categories:translation annotations_creators:machine-generated language_creators:found multilinguality:translation size_categories:unknown source_datasets:original language:zh license:cc-by-nc-sa-4.0 conditional-text-generation region:us
{"annotations_creators": ["machine-generated"], "language_creators": ["found"], "language": ["zh"], "license": ["cc-by-nc-sa-4.0"], "multilinguality": ["translation"], "size_categories": ["unknown"], "source_datasets": ["original"], "task_categories": ["text2text-generation", "translation"], "task_ids": [], "pretty_name": "Cantonese - Mandarin Translations", "language_bcp47": ["zh-CN", "zh-HK"], "tags": ["conditional-text-generation"]}
TAGS
#task_categories-text2text-generation #task_categories-translation #annotations_creators-machine-generated #language_creators-found #multilinguality-translation #size_categories-unknown #source_datasets-original #language-Chinese #license-cc-by-nc-sa-4.0 #conditional-text-generation #region-us 


# Dataset Card for cantonese-mandarin-translations

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage:
- Repository:
- Paper:
- Leaderboard:
- Point of Contact:

### Dataset Summary

This is a machine-translated parallel corpus between Cantonese (a Chinese dialect that is mainly spoken by Guangdong (province of China), Hong Kong, Macau and part of Malaysia) and Chinese (written form, in Simplified Chinese).
 

### Supported Tasks and Leaderboards

N/A

### Languages

- Cantonese ('yue')
- Simplified Chinese ('zh-CN')

## Dataset Structure

JSON lines with 'yue' field and 'zh' field for the parallel corpus.

### Data Instances

N/A

### Data Fields

- 'yue': Cantonese corpus
- 'zh': translated Chinese corpus

### Data Splits

No data splitting is done as of yet.

## Dataset Creation

The dataset is produced by doing the following:

- Download HKCancor Cantonese Corpus and CommonVoice Cantonese (Hong Kong Chinese 'yue') text corpus
- Extract text corpus and merge datasets
- Run text against Microsoft's Translator API from 'yue' to 'zh-Hans'

### Curation Rationale

Currently no such corpus exists, and it is hard to find such a corpus, so we tried to generate a reasonable batch of samples using machine translation for research purposes.

### Source Data

- HKCancor
- CommonVoice 7.0 Chinese (Hong Kong)

#### Initial Data Collection and Normalization

Normalization scripts will be included soon.

#### Who are the source language producers?

- HKCancor
- CommonVoice 7.0 Chinese (Hong Kong)

### Annotations

#### Annotation process

We run the Cantonese text corpus against Microsoft's Translator API.

#### Who are the annotators?

- Microsoft's Translator API

### Personal and Sensitive Information

N/A

## Considerations for Using the Data

### Social Impact of Dataset

We would like to share this parallel corpus and welcome contributions to preserve the Cantonese dialect.

### Discussion of Biases

N/A

### Other Known Limitations

This parallel corpus is machine-translated, it is not 100% accurate.

## Additional Information

### Dataset Curators

- Botisan AI
- Haoran (Simon) Liang

### Licensing Information

CC BY-NC-SA 4.0





### Contributions

Thanks to @lhr0909 for adding this dataset.
# Dataset Card for cantonese-mandarin-translations ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage:
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: ### Dataset Summary

This is a machine-translated parallel corpus between Cantonese (a Chinese dialect that is mainly spoken by Guangdong (province of China), Hong Kong, Macau and part of Malaysia) and Chinese (written form, in Simplified Chinese). ### Supported Tasks and Leaderboards

N/A ### Languages

- Cantonese ('yue')
- Simplified Chinese ('zh-CN') ## Dataset Structure

JSON lines with 'yue' field and 'zh' field for the parallel corpus. ### Data Instances

N/A ### Data Fields

- 'yue': Cantonese corpus
- 'zh': translated Chinese corpus ### Data Splits

No data splitting is done as of yet. ## Dataset Creation

The dataset is produced by doing the following:

- Download HKCancor Cantonese Corpus and CommonVoice Cantonese (Hong Kong Chinese 'yue') text corpus
- Extract text corpus and merge datasets
- Run text against Microsoft's Translator API from 'yue' to 'zh-Hans' ### Curation Rationale

Currently no such corpus exists, and it is hard to find such a corpus, so we tried to generate a reasonable batch of samples using machine translation for research purposes. ### Source Data

- HKCancor
- CommonVoice 7.0 Chinese (Hong Kong) #### Initial Data Collection and Normalization

Normalization scripts will be included soon. #### Who are the source language producers?

- HKCancor
- CommonVoice 7.0 Chinese (Hong Kong) ### Annotations #### Annotation process

We run the Cantonese text corpus against Microsoft's Translator API. #### Who are the annotators?

- Microsoft's Translator API ### Personal and Sensitive Information

N/A ## Considerations for Using the Data ### Social Impact of Dataset

We would like to share this parallel corpus and welcome contributions to preserve the Cantonese dialect. ### Discussion of Biases

N/A ### Other Known Limitations

This parallel corpus is machine-translated, it is not 100% accurate. ## Additional Information ### Dataset Curators

- Botisan AI
- Haoran (Simon) Liang ### Licensing Information

CC BY-NC-SA 4.0 ### Contributions

Thanks to @lhr0909 for adding this dataset.
TAGS
#task_categories-text2text-generation #task_categories-translation #annotations_creators-machine-generated #language_creators-found #multilinguality-translation #size_categories-unknown #source_datasets-original #language-Chinese #license-cc-by-nc-sa-4.0 #conditional-text-generation #region-us 
 # Dataset Card for cantonese-mandarin-translations ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage:
- Repository:
- Paper:
- Leaderboard:
- Point of Contact: ### Dataset Summary

This is a machine-translated parallel corpus between Cantonese (a Chinese dialect that is mainly spoken by Guangdong (province of China), Hong Kong, Macau and part of Malaysia) and Chinese (written form, in Simplified Chinese). ### Supported Tasks and Leaderboards

N/A ### Languages

- Cantonese ('yue')
- Simplified Chinese ('zh-CN') ## Dataset Structure

JSON lines with 'yue' field and 'zh' field for the parallel corpus. ### Data Instances

N/A ### Data Fields

- 'yue': Cantonese corpus
- 'zh': translated Chinese corpus ### Data Splits

No data splitting is done as of yet. ## Dataset Creation

The dataset is produced by doing the following:

- Download HKCancor Cantonese Corpus and CommonVoice Cantonese (Hong Kong Chinese 'yue') text corpus
- Extract text corpus and merge datasets
- Run text against Microsoft's Translator API from 'yue' to 'zh-Hans' ### Curation Rationale

Currently no such corpus exists, and it is hard to find such a corpus, so we tried to generate a reasonable batch of samples using machine translation for research purposes. ### Source Data

- HKCancor
- CommonVoice 7.0 Chinese (Hong Kong) #### Initial Data Collection and Normalization

Normalization scripts will be included soon. #### Who are the source language producers?

- HKCancor
- CommonVoice 7.0 Chinese (Hong Kong) ### Annotations #### Annotation process

We run the Cantonese text corpus against Microsoft's Translator API. #### Who are the annotators?

- Microsoft's Translator API ### Personal and Sensitive Information

N/A ## Considerations for Using the Data ### Social Impact of Dataset

We would like to share this parallel corpus and welcome contributions to preserve the Cantonese dialect. ### Discussion of Biases

N/A ### Other Known Limitations

This parallel corpus is machine-translated, it is not 100% accurate. ## Additional Information ### Dataset Curators

- Botisan AI
- Haoran (Simon) Liang ### Licensing Information

CC BY-NC-SA 4.0 ### Contributions

Thanks to @lhr0909 for adding this dataset.