text
🔥 77.2% on MMLU with 3.7B parameters 🚀
🔥 77.2% on MMLU with 3.7B parameters 🚀
new_line
new_line
text
... 3.7B active parameters, 40B in total parameters 📊
... 3.7B active parameters, 40B in total parameters 📊
new_line
new_line
text
7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉
7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉
new_line
new_line
text
Exciting enough? 😲
Exciting enough? 😲
new_line
new_line
text
That's Yuan2-M32 for you, released by IEIT-Yuan.
That's Yuan2-M32 for you, released by IEIT-Yuan.
new_line
text
A new 40B Mixture of Experts using a new Attention Router mechanism 🧠
A new 40B Mixture of Experts using a new Attention Router mechanism 🧠
new_line
new_line
text
32 experts with 2 active in generation ✌️
32 experts with 2 active in generation ✌️
new_line
new_line
text
8,192 context length 📝
8,192 context length 📝
new_line
new_line
text
Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.
Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.
new_line
new_line
text
Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.
Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.
new_line
new_line
text
Open-source - Apache 2.0 📜
Open-source - Apache 2.0 📜
new_line
new_line
text
Vocabulary size of 135,040 🗣️
Vocabulary size of 135,040 🗣️
new_line
new_line
text
Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸
Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸
new_line
new_line
text
Models: 
Models: 
link
https://huggingface.co/IEITYuan
https://huggingface.co/IEITYuan
text
 🌐
 🌐
new_line
text
Paper: 
Paper: 
resource
https://huggingface.co/papers/2405.17976
paper
2405.17976
https://huggingface.co/papers/2405.17976
Yuan 2.0-M32: Mixture of Experts with Attention Router (2405.17976)
text
 📄
 📄
🔥 77.2% on MMLU with 3.7B parameters 🚀

... 3.7B active parameters, 40B in total parameters 📊

7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉

Exciting enough? 😲

That's Yuan2-M32 for you, released by IEIT-Yuan.
A new 40B Mixture of Experts using a new Attention Router mechanism 🧠

32 experts with 2 active in generation ✌️

8,192 context length 📝

Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.

Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.

Open-source - Apache 2.0 📜

Vocabulary size of 135,040 🗣️

Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸

Models: https://huggingface.co/IEITYuan 🌐
Paper: https://huggingface.co/papers/2405.17976 📄