text
👋 Hi there!
👋 Hi there!
new_line
new_line
text
This is my very first post. 
This is my very first post. 
new_line
new_line
text
I'll use  it to share some old news: a math preference dataset for DPO!
I'll use  it to share some old news: a math preference dataset for DPO!
new_line
new_line
text
I created this dataset some time ago while we were developing distilabel (
I created this dataset some time ago while we were developing distilabel (
link
https://github.com/argilla-io/distilabel
https://github.com/argilla-io/distilabel
text
).
).
new_line
new_line
text
Some days ago we found out people are actually using it! So I'll use this post to explain how I built it in case it's useful for the community.
Some days ago we found out people are actually using it! So I'll use this post to explain how I built it in case it's useful for the community.
new_line
new_line
text
1. I used distilabel's SelfInstruct-inspired task to generate instructions about different math topics. I curated the instructions with Argilla (on Spaces!).
1. I used distilabel's SelfInstruct-inspired task to generate instructions about different math topics. I curated the instructions with Argilla (on Spaces!).
new_line
text
2. Then I used a distilabel Pipeline to build a preference dataset using gpt3.5 as generator and gpt4 as labeller. If I recall correctly I used our JudgeLM implementation (see 
2. Then I used a distilabel Pipeline to build a preference dataset using gpt3.5 as generator and gpt4 as labeller. If I recall correctly I used our JudgeLM implementation (see 
link
https://distilabel.argilla.io/latest/technical-reference/tasks/#judgelmtask
https://distilabel.argilla.io/latest/technical-reference/tasks/#judgelmtask
text
)
)
new_line
new_line
text
(see the screenshot with the dataset in the Argilla UI)
(see the screenshot with the dataset in the Argilla UI)
new_line
new_line
text
3. Then I just binarized into chosen, rejected pairs and voilà:
3. Then I just binarized into chosen, rejected pairs and voilà:
new_line
new_line
resource
https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo
dataset
argilla/distilabel-math-preference-dpo
https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo
new_line
new_line
text
The funny thing is that I used this to do a second DPO run over Notus-7B. I hoped to see an improvement on math/reasoning skills but it actually improved in STEM and Humanities and did worse on Math 🤣 . 
The funny thing is that I used this to do a second DPO run over Notus-7B. I hoped to see an improvement on math/reasoning skills but it actually improved in STEM and Humanities and did worse on Math 🤣 . 
new_line
new_line
text
In conclusion, this dataset was only a quick experiement. I'm happy to see the community found it useful. Data for DPO and fine-tuning are still a mystery, let's unveil these mysteries in 2024 together!
In conclusion, this dataset was only a quick experiement. I'm happy to see the community found it useful. Data for DPO and fine-tuning are still a mystery, let's unveil these mysteries in 2024 together!
new_line
new_line
text
Follow me for the most exciting datasets for LLMs (and maybe some great, small, efficient models). I plan to announce all Argilla open-source work here!
Follow me for the most exciting datasets for LLMs (and maybe some great, small, efficient models). I plan to announce all Argilla open-source work here!
👋 Hi there!

This is my very first post. 

I'll use  it to share some old news: a math preference dataset for DPO!

I created this dataset some time ago while we were developing distilabel (https://github.com/argilla-io/distilabel).

Some days ago we found out people are actually using it! So I'll use this post to explain how I built it in case it's useful for the community.

1. I used distilabel's SelfInstruct-inspired task to generate instructions about different math topics. I curated the instructions with Argilla (on Spaces!).
2. Then I used a distilabel Pipeline to build a preference dataset using gpt3.5 as generator and gpt4 as labeller. If I recall correctly I used our JudgeLM implementation (see https://distilabel.argilla.io/latest/technical-reference/tasks/#judgelmtask)

(see the screenshot with the dataset in the Argilla UI)

3. Then I just binarized into chosen, rejected pairs and voilà:

https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo

The funny thing is that I used this to do a second DPO run over Notus-7B. I hoped to see an improvement on math/reasoning skills but it actually improved in STEM and Humanities and did worse on Math 🤣 . 

In conclusion, this dataset was only a quick experiement. I'm happy to see the community found it useful. Data for DPO and fine-tuning are still a mystery, let's unveil these mysteries in 2024 together!

Follow me for the most exciting datasets for LLMs (and maybe some great, small, efficient models). I plan to announce all Argilla open-source work here!
👍
pacoid sparkycollier osanseviero gabrielmbmb mishig severo sugatoray davanstrien mlabonne qJakc MoritzLaurer KnutJaegersberg radames victor clem julien-c merve tmnam20 Erland DarkLord7771 abidlabs Masa-Erland
❤️
pcuenq osanseviero gabrielmbmb mishig plaguss davanstrien pierrci qJakc thomwolf Manel-Hik victor radames mariagrandury clem julien-c merve sbrandeis
🤗
qJakc Tonic mariagrandury julien-c davanstrien merve
https://cdn-avatars.huggingface.co/v1/production/uploads/60420dccc15e823a685f2b03/Dn7QTyy9SZ7jKN6xpufVD.png
Daniel Vila
dvilasuero
user
https://cdn-avatars.huggingface.co/v1/production/uploads/62a3bb1cd0d8c2c2169f0b88/eT2TS0IlQbZtz-F_zHLz9.jpeg
Joseph [open/acc] Pollack
Tonic
user