text
Introducing Zephyr 141B-A35B 🪁: 
Introducing Zephyr 141B-A35B 🪁: 
new_line
new_line
resource
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
model
HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
new_line
new_line
text
Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: 
Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: 
resource
https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1
model
mistral-community/Mixtral-8x22B-v0.1
https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1
new_line
new_line
text
Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:
Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:
new_line
new_line
text
🧑‍🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by 
🧑‍🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by 
mention
@JW17
JW17
text
 and 
 and 
mention
@nlee-208
nlee-208
text
 and 
 and 
mention
@j6mes
j6mes
text
 and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO. 
 and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO. 
new_line
new_line
text
🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from 
🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from 
mention
@LDJnr
LDJnr
text
resource
https://huggingface.co/datasets/LDJnr/Capybara
dataset
LDJnr/Capybara
https://huggingface.co/datasets/LDJnr/Capybara
text
 and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.
 and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.
new_line
new_line
text
What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.
What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.
new_line
new_line
text
Kudos to 
Kudos to 
mention
@alvarobartt
alvarobartt
text
mention
@JW17
JW17
text
 and 
 and 
mention
@nlee-208
nlee-208
text
 for this very nice and fast-paced collab!
 for this very nice and fast-paced collab!
new_line
new_line
text
For more details on the paper and dataset, checkout our collection: 
For more details on the paper and dataset, checkout our collection: 
resource
https://huggingface.co/collections/HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
collection
HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
https://huggingface.co/collections/HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
Introducing Zephyr 141B-A35B 🪁: 

https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1

Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1

Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:

🧑‍🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by @JW17 and @nlee-208 and @j6mes and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO. 

🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from @LDJnr  https://huggingface.co/datasets/LDJnr/Capybara and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.

What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.

Kudos to @alvarobartt @JW17 and @nlee-208 for this very nice and fast-paced collab!

For more details on the paper and dataset, checkout our collection: https://huggingface.co/collections/HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
❤️
alvarobartt mehungrymunchie nlee-208 SilentWraith JW17 samusenps hakunamatata1997 dakshdeephere NouRed victor osanseviero avinash02 fynnkroeger dillfrescott clem philschmid srisree rafaelsandroni not-lain netynet
🤗
beta3 clem SilentWraith JW17 mathiasn1 victor osanseviero NickyNicky alvarobartt dillfrescott philschmid not-lain