text
Could more DPO-style preference data be crucial for enhancing open LLMs across different languages? 
Could more DPO-style preference data be crucial for enhancing open LLMs across different languages? 
new_line
new_line
text
Leveraging a 7k preference dataset Argilla (
Leveraging a 7k preference dataset Argilla (
mention
@alvarobartt
alvarobartt
text
), Hugging Face (
), Hugging Face (
mention
@lewtun
lewtun
text
) and Kaist AI (
) and Kaist AI (
mention
@JW17
JW17
text
 & 
 & 
mention
@nlee-208
nlee-208
text
)
)
new_line
text
utilized Kaist AI's recently introduced ORPO technique 
utilized Kaist AI's recently introduced ORPO technique 
resource
https://huggingface.co/papers/2403.07691
paper
2403.07691
https://huggingface.co/papers/2403.07691
ORPO: Monolithic Preference Optimization without Reference Model (2403.07691)
text
 with the latest MistralAI MOE model to create a very high-performing open LLM: 
 with the latest MistralAI MOE model to create a very high-performing open LLM: 
resource
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
model
HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
new_line
new_line
text
Since ORPO doesn't require a separate SFT stage, all that is needed is a strong base model + high-quality DPO-style datasets.
Since ORPO doesn't require a separate SFT stage, all that is needed is a strong base model + high-quality DPO-style datasets.
new_line
new_line
text
Currently, there is a significant lack of non-English DPO datasets. Filling this gap could significantly improve open LLMs in various languages.
Currently, there is a significant lack of non-English DPO datasets. Filling this gap could significantly improve open LLMs in various languages.
new_line
new_line
text
You can get an overview of the current state of DPO datasets across different languages here: 
You can get an overview of the current state of DPO datasets across different languages here: 
link
https://huggingface.co/spaces/DIBT/preference_data_by_language
https://huggingface.co/spaces/DIBT/preference_data_by_language
text
Could more DPO-style preference data be crucial for enhancing open LLMs across different languages? 

Leveraging a 7k preference dataset Argilla (@alvarobartt), Hugging Face (@lewtun) and Kaist AI (@JW17 & @nlee-208)
utilized Kaist AI's recently introduced ORPO technique https://huggingface.co/papers/2403.07691 with the latest MistralAI MOE model to create a very high-performing open LLM: https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1

Since ORPO doesn't require a separate SFT stage, all that is needed is a strong base model + high-quality DPO-style datasets.

Currently, there is a significant lack of non-English DPO datasets. Filling this gap could significantly improve open LLMs in various languages.

You can get an overview of the current state of DPO datasets across different languages here: https://huggingface.co/spaces/DIBT/preference_data_by_language 
🚀
JW17 alvarobartt osanseviero nlee-208 NMontanaBrown Violette Heng666 raincandy-u DataSoul