text
🔥🆕🆕🔥 Dataset Drop: 4 KTO signal transformed versions of the highly loved Argilla DPO datasets.
🔥🆕🆕🔥 Dataset Drop: 4 KTO signal transformed versions of the highly loved Argilla DPO datasets.
new_line
new_line
text
KTO formats for:
KTO formats for:
new_line
text
- UltraFeedback Cleaned Binarized
- UltraFeedback Cleaned Binarized
new_line
text
- Distilabel Intel Orca
- Distilabel Intel Orca
new_line
text
- Distilabel Capybara
- Distilabel Capybara
new_line
text
- DPO mix
- DPO mix
new_line
new_line
resource
https://huggingface.co/collections/argilla/preference-datasets-for-kto-65f98314d7c1b04ab54d41a7
collection
argilla/preference-datasets-for-kto-65f98314d7c1b04ab54d41a7
https://huggingface.co/collections/argilla/preference-datasets-for-kto-65f98314d7c1b04ab54d41a7
new_line
new_line
text
Paper claims :) 
Paper claims :) 
new_line
new_line
link
https://arxiv.org/abs/2402.01306
https://arxiv.org/abs/2402.01306
new_line
new_line
text
KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of n DPO pairs and breaking it up into 2n examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal.
KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of n DPO pairs and breaking it up into 2n examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal.
new_line
new_line
text
KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset.
KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset.
new_line
new_line
text
When the pretrained model is sufficiently good, one can skip supervised finetuning and go straight to KTO without a loss in generation quality. In contrast, we find that without doing SFT first, DPO-aligned models are significantly worse at all scales.
When the pretrained model is sufficiently good, one can skip supervised finetuning and go straight to KTO without a loss in generation quality. In contrast, we find that without doing SFT first, DPO-aligned models are significantly worse at all scales.
new_line
new_line
text
Do you need something custom? Take a look at 
Do you need something custom? Take a look at 
mention
@davanstrien
davanstrien
text
 his guide on creating your own KTO dataset with Argilla and our community.
 his guide on creating your own KTO dataset with Argilla and our community.
new_line
new_line
link
https://github.com/huggingface/data-is-better-together/tree/main/kto-preference
https://github.com/huggingface/data-is-better-together/tree/main/kto-preference
🔥🆕🆕🔥 Dataset Drop: 4 KTO signal transformed versions of the highly loved Argilla DPO datasets.

KTO formats for:
- UltraFeedback Cleaned Binarized
- Distilabel Intel Orca
- Distilabel Capybara
- DPO mix

https://huggingface.co/collections/argilla/preference-datasets-for-kto-65f98314d7c1b04ab54d41a7

Paper claims :) 

https://arxiv.org/abs/2402.01306

KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of n DPO pairs and breaking it up into 2n examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal.

KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset.

When the pretrained model is sufficiently good, one can skip supervised finetuning and go straight to KTO without a loss in generation quality. In contrast, we find that without doing SFT first, DPO-aligned models are significantly worse at all scales.

Do you need something custom? Take a look at @davanstrien his guide on creating your own KTO dataset with Argilla and our community.

https://github.com/huggingface/data-is-better-together/tree/main/kto-preference
🔥
davanstrien ZennyKenny osanseviero giux78 clem
❤️
davanstrien samusenps osanseviero clem