text
🚀🧙🏼‍♂️Introducing OpenHermesPreferences: the largest open AI feedback dataset for RLHF & DPO
🚀🧙🏼‍♂️Introducing OpenHermesPreferences: the largest open AI feedback dataset for RLHF & DPO
new_line
new_line
text
> Using LLMs to improve other LLMs, at scale!
> Using LLMs to improve other LLMs, at scale!
new_line
new_line
text
Built in collaboration with the H4 Hugging Face team, it's a 1M preferences dataset on top of the amazing 
Built in collaboration with the H4 Hugging Face team, it's a 1M preferences dataset on top of the amazing 
mention
@teknium
teknium
text
 's dataset.
 's dataset.
new_line
new_line
text
Dataset:
Dataset:
new_line
resource
https://huggingface.co/datasets/argilla/OpenHermesPreferences
dataset
argilla/OpenHermesPreferences
https://huggingface.co/datasets/argilla/OpenHermesPreferences
new_line
new_line
text
The dataset is another example of open collaboration:
The dataset is another example of open collaboration:
new_line
new_line
text
> The H4 team created responses with Mixtral using llm-swarm
> The H4 team created responses with Mixtral using llm-swarm
new_line
new_line
text
> Argilla created responses with NousResearch Hermes-2-Yi-34B using distilabel
> Argilla created responses with NousResearch Hermes-2-Yi-34B using distilabel
new_line
new_line
text
> The H4 ranked these responses + original response with PairRM from AllenAI, University of Southern California, Zhejiang University (
> The H4 ranked these responses + original response with PairRM from AllenAI, University of Southern California, Zhejiang University (
mention
@yuchenlin
yuchenlin
text
mention
@DongfuTingle
DongfuTingle
text
 and colleagues)
 and colleagues)
new_line
new_line
text
We hope this dataset will help the community's research efforts towards understanding the role of AI feedback for LLM alignment.
We hope this dataset will help the community's research efforts towards understanding the role of AI feedback for LLM alignment.
new_line
new_line
text
We're particularly excited about the ability of filtering specific subsets to improve LLM skills like math or reasoning.
We're particularly excited about the ability of filtering specific subsets to improve LLM skills like math or reasoning.
new_line
new_line
text
Here's how easy it is to filter by subset:
Here's how easy it is to filter by subset:
new_line
new_line
code_fence
```
ds = load_dataset("HuggingFaceH4/OpenHermesPreferences", split="train")

# Get the categories of the source dataset
# ['airoboros2.2', 'CamelAI', 'caseus_custom', ...]
sources = ds.unique("source")

# Filter for a subset
ds_filtered = ds.filter(lambda x : x["source"] in ["metamath", "EvolInstruct_70k"], num_proc=6)
```
ds = load_dataset("HuggingFaceH4/OpenHermesPreferences", split="train")

# Get the categories of the source dataset
# ['airoboros2.2', 'CamelAI', 'caseus_custom', ...]
sources = ds.unique("source")

# Filter for a subset
ds_filtered = ds.filter(lambda x : x["source"] in ["metamath", "EvolInstruct_70k"], num_proc=6)
new_line
new_line
text
As usual, all the scripts to reproduce this work are available and open to the community!
As usual, all the scripts to reproduce this work are available and open to the community!
new_line
new_line
resource
https://huggingface.co/datasets/argilla/OpenHermesPreferences/tree/main
dataset
argilla/OpenHermesPreferences
https://huggingface.co/datasets/argilla/OpenHermesPreferences/tree/main
new_line
new_line
text
So fun collab between 
So fun collab between 
mention
@vwxyzjn
vwxyzjn
text
 , 
 , 
mention
@plaguss
plaguss
text
, 
, 
mention
@kashif
kashif
text
, 
, 
mention
@philschmid
philschmid
text
 &  
 &  
mention
@lewtun
lewtun
text
!
!
new_line
new_line
text
Open Source AI FTW! 
Open Source AI FTW! 
new_line
🚀🧙🏼‍♂️Introducing OpenHermesPreferences: the largest open AI feedback dataset for RLHF & DPO

> Using LLMs to improve other LLMs, at scale!

Built in collaboration with the H4 Hugging Face team, it's a 1M preferences dataset on top of the amazing @teknium 's dataset.

Dataset:
https://huggingface.co/datasets/argilla/OpenHermesPreferences

The dataset is another example of open collaboration:

> The H4 team created responses with Mixtral using llm-swarm

> Argilla created responses with NousResearch Hermes-2-Yi-34B using distilabel

> The H4 ranked these responses + original response with PairRM from AllenAI, University of Southern California, Zhejiang University (@yuchenlin @DongfuTingle and colleagues)

We hope this dataset will help the community's research efforts towards understanding the role of AI feedback for LLM alignment.

We're particularly excited about the ability of filtering specific subsets to improve LLM skills like math or reasoning.

Here's how easy it is to filter by subset:

```
ds = load_dataset("HuggingFaceH4/OpenHermesPreferences", split="train")

# Get the categories of the source dataset
# ['airoboros2.2', 'CamelAI', 'caseus_custom', ...]
sources = ds.unique("source")

# Filter for a subset
ds_filtered = ds.filter(lambda x : x["source"] in ["metamath", "EvolInstruct_70k"], num_proc=6)
```

As usual, all the scripts to reproduce this work are available and open to the community!

https://huggingface.co/datasets/argilla/OpenHermesPreferences/tree/main

So fun collab between @vwxyzjn , @plaguss, @kashif, @philschmid &  @lewtun!

Open Source AI FTW! 

❤️
davanstrien plaguss philschmid osanseviero alielfilali01 Rybens lewtun xiaotianhan gabrielmbmb clem vladbogo koutch kramp thivav Nindaleth
🤯
philschmid osanseviero alielfilali01 clem
🤗
alielfilali01 clem
https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg
Daniel van Strien
davanstrien
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
Lewis Tunstall
lewtun
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857146757-5e67bdd61009063689407479.jpeg
Clem 🤗
clem
user