# Dataset Card for "dolly_hhrlhf"

This dataset is a combination of [Databrick's dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) dataset and a filtered subset of [Anthropic's HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf). It also includes a test split, which was missing in the original `dolly` set. That test set is composed of 200 randomly selected samples from `dolly` + 4,929 of the test set samples from HH-RLHF which made it through the filtering process. The train set contains 59,310 samples; `15,014 - 200 = 14,814` from Dolly, and the remaining 44,496 from HH-RLHF.

It is slightly larger than Alpaca, and in our experience of slightly higher quality, but is usable for commercial purposes so long as you follow the terms of the license.

## Filtering process

As mentioned, the HH-RLHF data in this dataset is filtered. Specifically, we take the first turn of the convesation, then remove any samples where the assistant:

- uses the word "human", "thank", or "sorry"
- asks a question
- uses a first person pronoun

This leaves samples which look like instruction-following, as opposed to conversation.

## License/Attribution
<!--
**Copyright (2023) MosaicML, Inc.**
-->
This dataset was developed at MosaicML (https://www.mosaicml.com) and its use is subject to the CC BY-SA 3.0 license. 

Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license:

Wikipedia (various pages) - https://www.wikipedia.org/
Copyright © Wikipedia editors and contributors.

Databricks (https://www.databricks.com)
Copyright © Databricks

When citing this dataset, please use the following:

```
@misc{mosaicml2023dolly_hhrlhf,
  author = {MosaicML},
  title = {Dolly-HHRLHF Dataset},
  year = {2023},
  publisher = {HuggingFace Datasets},
  howpublished = {https://huggingface.co/datasets/mosaicml/dolly_hhrlhf},
}
```
task_categories:text-generation language:en license:cc-by-sa-3.0 region:us
{"language": ["en"], "license": "cc-by-sa-3.0", "task_categories": ["text-generation"], "pretty_name": "Dolly HH-RLHF", "dataset_info": {"features": [{"name": "prompt", "dtype": "string"}, {"name": "response", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 43781455.002688624, "num_examples": 59310}, {"name": "test", "num_bytes": 4479286.805304853, "num_examples": 5129}], "download_size": 24882010, "dataset_size": 48260741.80799348}}
TAGS
#task_categories-text-generation #language-English #license-cc-by-sa-3.0 #region-us 

# Dataset Card for "dolly_hhrlhf"

This dataset is a combination of Databrick's dolly-15k dataset and a filtered subset of Anthropic's HH-RLHF. It also includes a test split, which was missing in the original 'dolly' set. That test set is composed of 200 randomly selected samples from 'dolly' + 4,929 of the test set samples from HH-RLHF which made it through the filtering process. The train set contains 59,310 samples; '15,014 - 200 = 14,814' from Dolly, and the remaining 44,496 from HH-RLHF.

It is slightly larger than Alpaca, and in our experience of slightly higher quality, but is usable for commercial purposes so long as you follow the terms of the license.

## Filtering process

As mentioned, the HH-RLHF data in this dataset is filtered. Specifically, we take the first turn of the convesation, then remove any samples where the assistant:

- uses the word "human", "thank", or "sorry"
- asks a question
- uses a first person pronoun

This leaves samples which look like instruction-following, as opposed to conversation.

## License/Attribution

This dataset was developed at MosaicML (URL) and its use is subject to the CC BY-SA 3.0 license. 

Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license:

Wikipedia (various pages) - URL
Copyright © Wikipedia editors and contributors.

Databricks (URL)
Copyright © Databricks

When citing this dataset, please use the following:


# Dataset Card for "dolly_hhrlhf"

This dataset is a combination of Databrick's dolly-15k dataset and a filtered subset of Anthropic's HH-RLHF. It also includes a test split, which was missing in the original 'dolly' set. That test set is composed of 200 randomly selected samples from 'dolly' + 4,929 of the test set samples from HH-RLHF which made it through the filtering process. The train set contains 59,310 samples; '15,014 - 200 = 14,814' from Dolly, and the remaining 44,496 from HH-RLHF.

It is slightly larger than Alpaca, and in our experience of slightly higher quality, but is usable for commercial purposes so long as you follow the terms of the license. ## Filtering process

As mentioned, the HH-RLHF data in this dataset is filtered. Specifically, we take the first turn of the convesation, then remove any samples where the assistant:

- uses the word "human", "thank", or "sorry"
- asks a question
- uses a first person pronoun

This leaves samples which look like instruction-following, as opposed to conversation. ## License/Attribution

This dataset was developed at MosaicML (URL) and its use is subject to the CC BY-SA 3.0 license. 

Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license:

Wikipedia (various pages) - URL
Copyright © Wikipedia editors and contributors.

Databricks (URL)
Copyright © Databricks

When citing this dataset, please use the following:
TAGS
#task_categories-text-generation #language-English #license-cc-by-sa-3.0 #region-us 
 # Dataset Card for "dolly_hhrlhf"

This dataset is a combination of Databrick's dolly-15k dataset and a filtered subset of Anthropic's HH-RLHF. It also includes a test split, which was missing in the original 'dolly' set. That test set is composed of 200 randomly selected samples from 'dolly' + 4,929 of the test set samples from HH-RLHF which made it through the filtering process. The train set contains 59,310 samples; '15,014 - 200 = 14,814' from Dolly, and the remaining 44,496 from HH-RLHF.

It is slightly larger than Alpaca, and in our experience of slightly higher quality, but is usable for commercial purposes so long as you follow the terms of the license. ## Filtering process

As mentioned, the HH-RLHF data in this dataset is filtered. Specifically, we take the first turn of the convesation, then remove any samples where the assistant:

- uses the word "human", "thank", or "sorry"
- asks a question
- uses a first person pronoun

This leaves samples which look like instruction-following, as opposed to conversation. ## License/Attribution

This dataset was developed at MosaicML (URL) and its use is subject to the CC BY-SA 3.0 license. 

Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license:

Wikipedia (various pages) - URL
Copyright © Wikipedia editors and contributors.

Databricks (URL)
Copyright © Databricks

When citing this dataset, please use the following: