license:mit region:us
---
license: mit
---

# The Hermes_preference dataset

<!-- Provide a quick summary of the dataset. -->

The **Hermes_preference** dataset is the type of feedback dataset, used for training reward models which is used for RLHF! 
In addition, **Hermes_preference** dataset can be also used for DPO!
We collect the preference data from several popular feedback datasets([UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback), [hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf), [rlhf-reward-datasets](https://huggingface.co/datasets/yitingxie/rlhf-reward-datasets)) through sampling and preprocessing.
As a result, we could have collected approximately 190K preference data.

To collect high-quality feedback data, we decided to collect feedback data from [UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) & [rlhf-reward-datasets](https://huggingface.co/datasets/yitingxie/rlhf-reward-datasets) which are curated datasets.
In addition, we also collect the data from [hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) to accumulate the data that teach the models to output helpful and harmless response.
We hope that **Hermes_preference** dataset provides a promising way to future RLHF & DPO research!


## Dataset Details

<!-- Provide a longer summary of what this dataset is. -->

The **Hermes_preference** dataset is a mixture of several popular preference datasets(UltraFeedback, hh-rlhf, rlhf-reward-datasets) as we mentioned above.
The purpose of this dataset is to make a preference dataset that consists of more varied data.
To accomplish this purpose, we selected the UltraFeedback, hh-rlhf, and rlhf-reward-datasets as the base dataset.
More specifically, we sampled and preprocessed the datasets mentioned above to make Hermes_preference dataset more structural.



- **Curated by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** en
- **License:** MIT

### Source Data

The Hermes_preference dataset consists of the following datasets.

- [**openbmb/UltraFeedback**](https://huggingface.co/datasets/openbmb/UltraFeedback)
- [**Anthropic/hh-rlhf**](https://huggingface.co/datasets/Anthropic/hh-rlhf)
- [**yitingxie/rlhf-reward-datasets**](https://huggingface.co/datasets/yitingxie/rlhf-reward-datasets)
<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

### Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

- **Repository:** [More Information Needed]
- **Model** [More Information Needed]

## Dataset Structure

The structure of **Hermes_prference** dataset is as follows:

```
{
    "source": The source dataset of data,
    "prompt": The instruction of question,
    "chosen": Choosed response,
    "rejected": Rejected response
}
```
<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

## Dataset Creation

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

## Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Dataset Card Authors [optional]

[More Information Needed]

## Dataset Card Contact

[More Information Needed]