
# Dataset Card for Quora Chat Dutch

## Dataset Description

- **Homepage:** N/A
- **Repository:** N/A
- **Paper:** N/A
- **Leaderboard:** N/A
- **Point of Contact:** Bram Vanroy

### Dataset Summary

**Update 15/1/2024**: the format of this dataset has been updated so that it now follows the `messages` format, where each conversation is a list of messages (dictionaries) and each dictionary has a role (user, assistant or system) and content key. Importantly, this dataset was also filtered by its language identification tag. We ran LID with [FastText](https://huggingface.co/facebook/fasttext-language-identification) (results are in the separate `3-lid`  branch) and filtered on Dutch only. Therefore, the current version contains fewer samples than the original version. Note that this may mean that translation-related tasks or even code questions have been filtered out! Finally, we have made a train/test split (`train_sft` and `test_sft`), following the approach by Hugging Face in e.g. [UltraChat 200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k). The original version of the dataset is available in branch `1-original`.

This dataset contains 48,761 (43,919 train, 4842 test) conversations between een AI assistant and a (fake) "Human" (generated) in Dutch. They are translations of [Baize](https://github.com/project-baize/baize-chatbot/tree/6790946f638d60fcaf397574189124f15792f35a/data)'s machine-generated answers to the [Quora](https://huggingface.co/datasets/quora) dataset. 

☕ [**Want to help me out?**](https://www.buymeacoffee.com/bramvanroy) Translating the data with the OpenAI API, and prompt testing, cost me 💸$135.65💸. If you like this dataset, please consider [buying me a coffee](https://www.buymeacoffee.com/bramvanroy) to offset a portion of this cost, I appreciate it a lot! ☕

If you use this dataset or refer to it, please use the following citation:

Vanroy, B. (2023). *Language Resources for Dutch Large Language Modelling*. [https://arxiv.org/abs/2312.12852](https://arxiv.org/abs/2312.12852)

```bibtext
@article{vanroy2023language,
  title={Language Resources for {Dutch} Large Language Modelling},
  author={Vanroy, Bram},
  journal={arXiv preprint arXiv:2312.12852},
  year={2023}
}
```


### Languages

- Dutch

## Dataset Structure

### Data Instances

```python
{
    'messages':[
       {'content': 'Ik wil weten hoe ik meer upvotes kan krijgen op mijn berichten.', 'role': 'user'},
       {'content': 'Interessante vraag! Heb je geprobeerd om lid te worden van communities of subreddits die gerelateerd zijn aan het onderwerp van je bericht?', 'role': 'assistant'},
       {'content': 'Ja, dat heb ik. Maar mijn berichten krijgen nog steeds niet veel upvotes. Heb je nog andere tips?', 'role': 'user'},
       {'content': 'Een ding dat je kunt doen is het creëren van hoogwaardige inhoud die interessant en relevant is voor het publiek van de subreddit. Bovendien kun je in contact komen met je publiek door te reageren op de berichten van andere mensen en te reageren op opmerkingen op je eigen berichten.', 'role': 'assistant'},
       {'content': 'Dat is een goed punt. Ik heb ook gehoord dat pakkende koppen en goede timing een verschil kunnen maken. Wat denk jij?', 'role': 'user'},
       {'content': 'Ja, een pakkende kop hebben en posten op een moment dat je publiek het meest actief is, kan zeker je kansen vergroten om upvotes te krijgen. Maar uiteindelijk komt het allemaal neer op het creëren van kwalitatieve inhoud die aantrekkelijk is voor je publiek.', 'role': 'assistant'},
       {'content': 'Ik snap het. Bedankt voor het advies! Heb je nog andere tips om de betrokkenheid bij mijn berichten te verbeteren?', 'role': 'user'},
       {'content': "Natuurlijk, je kunt proberen multimedia zoals afbeeldingen of video's te gebruiken om je bericht interessanter te maken, en ook overwegen om samen te werken met andere gebruikers of influencers om je bereik te vergroten. En vergeet niet om je berichten op andere sociale media platforms te promoten om meer kijkers aan te trekken.", 'role': 'assistant'}
    ],
   'prompt': 'Ik wil weten hoe ik meer upvotes kan krijgen op mijn berichten.',
   'prompt_id': 'a79a84a83cc11063aaa63becacdf4aaa91866a265e37818a4a9684014b7c85d9'
}
```

### Data Fields

- **prompt**: the initial user prompt
- **prompt_id**: the unique hash of the prompt
- **messages**: list of messages (dictionaries) where each dictionary has a role (user, assistant, or system) and content


## Dataset Creation

Both the translations and the topics were translated with OpenAI's API for `gpt-3.5-turbo`. `max_tokens=1024, temperature=0` as parameters.

The prompt template to translate the input is:

```python
CONVERSATION_TRANSLATION_PROMPT = """You are asked to translate a conversation between an AI assistant and a human from {src_lang} into {tgt_lang}.

Here are the requirements that you should adhere to:
1. maintain the format: the conversation consists of the AI (marked as `[|AI|]`) and the human (`[|Human|]`) talking in turns and responding to each other;
2. do not translate the speaker identifiers `[|AI|]` and `[|Human|]` but always copy them into the translation in appropriate places;
3. ensure accurate translation and keep the correctness of the conversation;
4. make sure that text is fluent to read and does not contain grammatical errors. Use standard {tgt_lang} without regional bias;
5. translate the human's text using informal, but standard, language;
6. make sure to avoid biases (such as gender bias, grammatical bias, social bias);
7. if the human asks to correct grammar mistakes or spelling mistakes then you have to generate a similar mistake in {tgt_lang}, and then also generate a corrected output version for the AI in {tgt_lang};
8. if the human asks to translate text from one to another language, then you only translate the human's question to {tgt_lang} but you keep the translation that the AI provides in the language that the human requested;
9. do not translate code fragments but copy them as they are. If there are English examples, variable names or definitions in code fragments, keep them in English.

Now translate the following conversation with the requirements set out above. Do not provide an explanation and do not add anything else.\n\n"""
```

The prompt to translate the topic is:

```python
TOPIC_TRANSLATION_PROMPT = "Translate the following title of a conversation from {src_lang} to {tgt_lang} in a succinct," \
                           " summarizing manner. Translate accurately and formally. Do not provide any explanation" \
                           " about the translation and do not include the original title.\n\n"
```

The system message was:

```
You are a helpful assistant that translates English to Dutch to the requirements that are given to you.
```

Note that 12 items (0.02%) were not successfully translated. The translation was missing the AI identifier `[|AI|]` and/or the human one `[|Human|]`. The IDs for the missing items are `[1352, 14599, 25985, 27167, 28254, 33731, 41550, 44364, 47293, 49060, 51383, 53740]`.

### Source Data

#### Initial Data Collection and Normalization

Initial data generation by [Baize](https://github.com/project-baize/baize-chatbot), which in turn relied on [Quora](https://huggingface.co/datasets/quora).

#### Who are the source language producers?

Quora users and OpenAI's GPT models (via [Baize](https://github.com/project-baize/baize-chatbot)).

## Considerations for Using the Data

Note that neither the translations in this new dataset, nor the machine-generated dialogues in the original [Baize](https://github.com/project-baize/baize-chatbot) dataset, have been verified by humans.

### Discussion of Biases

As with any machine-generated texts, users should be aware of potential biases that are included in this dataset. Although the prompt specifically includes `make sure to avoid biases (such as gender bias, grammatical bias, social bias)`, of course the impact of such command is not known. It is likely that biases remain in the dataset so use with caution.

### Other Known Limitations

The translation quality has not been verified. Use at your own risk!

### Licensing Information

This text was generated (either in part or in full) with GPT-3 (`gpt-3.5-turbo`), OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication. Therefore commercial usage is not allowed.

If you use this dataset, you must also follow the [Sharing](https://openai.com/policies/sharing-publication-policy) and [Usage](https://openai.com/policies/usage-policies) policies. 


### Contributions

Thanks to [Baize](https://github.com/project-baize/baize-chatbot) for the initial machine-generated dialogues, based on [quora](https://huggingface.co/datasets/quora).
task_categories:question-answering task_categories:text-generation size_categories:10K<n<100K language:nl license:cc-by-nc-4.0 baize quora chat arxiv:2312.12852 doi:10.57967/hf/0524 region:us
{"language": ["nl"], "license": "cc-by-nc-4.0", "size_categories": ["10K<n<100K"], "task_categories": ["question-answering", "text-generation"], "pretty_name": "Quora Chat Dutch", "tags": ["baize", "quora", "chat"], "dataset_info": {"features": [{"name": "prompt", "dtype": "string"}, {"name": "prompt_id", "dtype": "string"}, {"name": "messages", "list": [{"name": "content", "dtype": "string"}, {"name": "role", "dtype": "string"}]}], "splits": [{"name": "train_sft", "num_bytes": 67873239, "num_examples": 43919}, {"name": "test_sft", "num_bytes": 7472429, "num_examples": 4842}], "download_size": 41224462, "dataset_size": 75345668}, "configs": [{"config_name": "default", "data_files": [{"split": "train_sft", "path": "data/train_sft-*"}, {"split": "test_sft", "path": "data/test_sft-*"}]}]}
TAGS
#task_categories-question-answering #task_categories-text-generation #size_categories-10K<n<100K #language-Dutch #license-cc-by-nc-4.0 #baize #quora #chat #arxiv-2312.12852 #doi-10.57967/hf/0524 #region-us 


# Dataset Card for Quora Chat Dutch

## Dataset Description

- Homepage: N/A
- Repository: N/A
- Paper: N/A
- Leaderboard: N/A
- Point of Contact: Bram Vanroy

### Dataset Summary

Update 15/1/2024: the format of this dataset has been updated so that it now follows the 'messages' format, where each conversation is a list of messages (dictionaries) and each dictionary has a role (user, assistant or system) and content key. Importantly, this dataset was also filtered by its language identification tag. We ran LID with FastText (results are in the separate '3-lid'  branch) and filtered on Dutch only. Therefore, the current version contains fewer samples than the original version. Note that this may mean that translation-related tasks or even code questions have been filtered out! Finally, we have made a train/test split ('train_sft' and 'test_sft'), following the approach by Hugging Face in e.g. UltraChat 200k. The original version of the dataset is available in branch '1-original'.

This dataset contains 48,761 (43,919 train, 4842 test) conversations between een AI assistant and a (fake) "Human" (generated) in Dutch. They are translations of Baize's machine-generated answers to the Quora dataset. 

 Want to help me out? Translating the data with the OpenAI API, and prompt testing, cost me $135.65. If you like this dataset, please consider buying me a coffee to offset a portion of this cost, I appreciate it a lot! 

If you use this dataset or refer to it, please use the following citation:

Vanroy, B. (2023). *Language Resources for Dutch Large Language Modelling*. URL




### Languages

- Dutch

## Dataset Structure

### Data Instances



### Data Fields

- prompt: the initial user prompt
- prompt_id: the unique hash of the prompt
- messages: list of messages (dictionaries) where each dictionary has a role (user, assistant, or system) and content


## Dataset Creation

Both the translations and the topics were translated with OpenAI's API for 'gpt-3.5-turbo'. 'max_tokens=1024, temperature=0' as parameters.

The prompt template to translate the input is:



The prompt to translate the topic is:



The system message was:



Note that 12 items (0.02%) were not successfully translated. The translation was missing the AI identifier '[|AI|]' and/or the human one '[|Human|]'. The IDs for the missing items are '[1352, 14599, 25985, 27167, 28254, 33731, 41550, 44364, 47293, 49060, 51383, 53740]'.

### Source Data

#### Initial Data Collection and Normalization

Initial data generation by Baize, which in turn relied on Quora.

#### Who are the source language producers?

Quora users and OpenAI's GPT models (via Baize).

## Considerations for Using the Data

Note that neither the translations in this new dataset, nor the machine-generated dialogues in the original Baize dataset, have been verified by humans.

### Discussion of Biases

As with any machine-generated texts, users should be aware of potential biases that are included in this dataset. Although the prompt specifically includes 'make sure to avoid biases (such as gender bias, grammatical bias, social bias)', of course the impact of such command is not known. It is likely that biases remain in the dataset so use with caution.

### Other Known Limitations

The translation quality has not been verified. Use at your own risk!

### Licensing Information

This text was generated (either in part or in full) with GPT-3 ('gpt-3.5-turbo'), OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication. Therefore commercial usage is not allowed.

If you use this dataset, you must also follow the Sharing and Usage policies. 


### Contributions

Thanks to Baize for the initial machine-generated dialogues, based on quora.
# Dataset Card for Quora Chat Dutch ## Dataset Description

- Homepage: N/A
- Repository: N/A
- Paper: N/A
- Leaderboard: N/A
- Point of Contact: Bram Vanroy ### Dataset Summary

Update 15/1/2024: the format of this dataset has been updated so that it now follows the 'messages' format, where each conversation is a list of messages (dictionaries) and each dictionary has a role (user, assistant or system) and content key. Importantly, this dataset was also filtered by its language identification tag. We ran LID with FastText (results are in the separate '3-lid'  branch) and filtered on Dutch only. Therefore, the current version contains fewer samples than the original version. Note that this may mean that translation-related tasks or even code questions have been filtered out! Finally, we have made a train/test split ('train_sft' and 'test_sft'), following the approach by Hugging Face in e.g. UltraChat 200k. The original version of the dataset is available in branch '1-original'.

This dataset contains 48,761 (43,919 train, 4842 test) conversations between een AI assistant and a (fake) "Human" (generated) in Dutch. They are translations of Baize's machine-generated answers to the Quora dataset. 

 Want to help me out? Translating the data with the OpenAI API, and prompt testing, cost me $135.65. If you like this dataset, please consider buying me a coffee to offset a portion of this cost, I appreciate it a lot! 

If you use this dataset or refer to it, please use the following citation:

Vanroy, B. (2023). *Language Resources for Dutch Large Language Modelling*. URL ### Languages

- Dutch ## Dataset Structure ### Data Instances ### Data Fields

- prompt: the initial user prompt
- prompt_id: the unique hash of the prompt
- messages: list of messages (dictionaries) where each dictionary has a role (user, assistant, or system) and content ## Dataset Creation

Both the translations and the topics were translated with OpenAI's API for 'gpt-3.5-turbo'. 'max_tokens=1024, temperature=0' as parameters.

The prompt template to translate the input is:



The prompt to translate the topic is:



The system message was:



Note that 12 items (0.02%) were not successfully translated. The translation was missing the AI identifier '[|AI|]' and/or the human one '[|Human|]'. The IDs for the missing items are '[1352, 14599, 25985, 27167, 28254, 33731, 41550, 44364, 47293, 49060, 51383, 53740]'. ### Source Data #### Initial Data Collection and Normalization

Initial data generation by Baize, which in turn relied on Quora. #### Who are the source language producers?

Quora users and OpenAI's GPT models (via Baize). ## Considerations for Using the Data

Note that neither the translations in this new dataset, nor the machine-generated dialogues in the original Baize dataset, have been verified by humans. ### Discussion of Biases

As with any machine-generated texts, users should be aware of potential biases that are included in this dataset. Although the prompt specifically includes 'make sure to avoid biases (such as gender bias, grammatical bias, social bias)', of course the impact of such command is not known. It is likely that biases remain in the dataset so use with caution. ### Other Known Limitations

The translation quality has not been verified. Use at your own risk! ### Licensing Information

This text was generated (either in part or in full) with GPT-3 ('gpt-3.5-turbo'), OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication. Therefore commercial usage is not allowed.

If you use this dataset, you must also follow the Sharing and Usage policies. ### Contributions

Thanks to Baize for the initial machine-generated dialogues, based on quora.
TAGS
#task_categories-question-answering #task_categories-text-generation #size_categories-10K<n<100K #language-Dutch #license-cc-by-nc-4.0 #baize #quora #chat #arxiv-2312.12852 #doi-10.57967/hf/0524 #region-us 
 # Dataset Card for Quora Chat Dutch ## Dataset Description

- Homepage: N/A
- Repository: N/A
- Paper: N/A
- Leaderboard: N/A
- Point of Contact: Bram Vanroy ### Dataset Summary

Update 15/1/2024: the format of this dataset has been updated so that it now follows the 'messages' format, where each conversation is a list of messages (dictionaries) and each dictionary has a role (user, assistant or system) and content key. Importantly, this dataset was also filtered by its language identification tag. We ran LID with FastText (results are in the separate '3-lid'  branch) and filtered on Dutch only. Therefore, the current version contains fewer samples than the original version. Note that this may mean that translation-related tasks or even code questions have been filtered out! Finally, we have made a train/test split ('train_sft' and 'test_sft'), following the approach by Hugging Face in e.g. UltraChat 200k. The original version of the dataset is available in branch '1-original'.

This dataset contains 48,761 (43,919 train, 4842 test) conversations between een AI assistant and a (fake) "Human" (generated) in Dutch. They are translations of Baize's machine-generated answers to the Quora dataset. 

 Want to help me out? Translating the data with the OpenAI API, and prompt testing, cost me $135.65. If you like this dataset, please consider buying me a coffee to offset a portion of this cost, I appreciate it a lot! 

If you use this dataset or refer to it, please use the following citation:

Vanroy, B. (2023). *Language Resources for Dutch Large Language Modelling*. URL ### Languages

- Dutch ## Dataset Structure ### Data Instances ### Data Fields

- prompt: the initial user prompt
- prompt_id: the unique hash of the prompt
- messages: list of messages (dictionaries) where each dictionary has a role (user, assistant, or system) and content ## Dataset Creation

Both the translations and the topics were translated with OpenAI's API for 'gpt-3.5-turbo'. 'max_tokens=1024, temperature=0' as parameters.

The prompt template to translate the input is:



The prompt to translate the topic is:



The system message was:



Note that 12 items (0.02%) were not successfully translated. The translation was missing the AI identifier '[|AI|]' and/or the human one '[|Human|]'. The IDs for the missing items are '[1352, 14599, 25985, 27167, 28254, 33731, 41550, 44364, 47293, 49060, 51383, 53740]'. ### Source Data #### Initial Data Collection and Normalization

Initial data generation by Baize, which in turn relied on Quora. #### Who are the source language producers?

Quora users and OpenAI's GPT models (via Baize). ## Considerations for Using the Data

Note that neither the translations in this new dataset, nor the machine-generated dialogues in the original Baize dataset, have been verified by humans. ### Discussion of Biases

As with any machine-generated texts, users should be aware of potential biases that are included in this dataset. Although the prompt specifically includes 'make sure to avoid biases (such as gender bias, grammatical bias, social bias)', of course the impact of such command is not known. It is likely that biases remain in the dataset so use with caution. ### Other Known Limitations

The translation quality has not been verified. Use at your own risk! ### Licensing Information

This text was generated (either in part or in full) with GPT-3 ('gpt-3.5-turbo'), OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication. Therefore commercial usage is not allowed.

If you use this dataset, you must also follow the Sharing and Usage policies. ### Contributions

Thanks to Baize for the initial machine-generated dialogues, based on quora.