text
Very interesting model just released by MyShell: 
Very interesting model just released by MyShell: 
resource
https://huggingface.co/jetmoe/jetmoe-8b
model
jetmoe/jetmoe-8b
https://huggingface.co/jetmoe/jetmoe-8b
text
 . It's a 8B-parameters MoE LLM so 2.2B active parameters, really efficient.
 . It's a 8B-parameters MoE LLM so 2.2B active parameters, really efficient.
new_line
new_line
text
Main characteristics:
Main characteristics:
new_line
text
- impressive performances for its size (beating 
- impressive performances for its size (beating 
resource
https://huggingface.co/meta-llama/Llama-2-7b
model
meta-llama/Llama-2-7b
https://huggingface.co/meta-llama/Llama-2-7b
text
 and 
 and 
resource
https://huggingface.co/huggyllama/llama-13b
model
huggyllama/llama-13b
https://huggingface.co/huggyllama/llama-13b
text
)
)
new_line
text
- combine Mixture of Attention heads (MoA) and Mixture of MLP Experts (MoE) – 8 experts with 2 being active for each token
- combine Mixture of Attention heads (MoA) and Mixture of MLP Experts (MoE) – 8 experts with 2 being active for each token
new_line
text
- trained on a rather limited 1.25T tokens from publicly available datasets – training recipe follows the MiniCPM's two-phases training method => first time I see this for a 2B+ model
- trained on a rather limited 1.25T tokens from publicly available datasets – training recipe follows the MiniCPM's two-phases training method => first time I see this for a 2B+ model
new_line
text
- $100k to train
- $100k to train
new_line
text
- open weights - open sharing of recipes - open dataset - open code => ♡
- open weights - open sharing of recipes - open dataset - open code => ♡
new_line
text
- still interesting room to improve performances (be it only by training longer)
- still interesting room to improve performances (be it only by training longer)
new_line
new_line
text
Links:
Links:
new_line
text
- report: 
- report: 
link
https://research.myshell.ai/jetmoe
https://research.myshell.ai/jetmoe
new_line
text
- model: 
- model: 
resource
https://huggingface.co/jetmoe/jetmoe-8b
model
jetmoe/jetmoe-8b
https://huggingface.co/jetmoe/jetmoe-8b
new_line
text
- code: 
- code: 
link
https://github.com/myshell-ai/JetMoE
https://github.com/myshell-ai/JetMoE
new_line
new_line
text
Note: I actually detailed all of the MiniCPM schedule, Mixture-of-expert (MoE) and many of the datasets used in this work in my recent little guide to building LLMs in 2024, so feel free to check it out if you want to learn more on these topics: 
Note: I actually detailed all of the MiniCPM schedule, Mixture-of-expert (MoE) and many of the datasets used in this work in my recent little guide to building LLMs in 2024, so feel free to check it out if you want to learn more on these topics: 
link
https://www.youtube.com/watch?v=2-SPH9hIKT8
https://www.youtube.com/watch?v=2-SPH9hIKT8
Very interesting model just released by MyShell: https://huggingface.co/jetmoe/jetmoe-8b . It's a 8B-parameters MoE LLM so 2.2B active parameters, really efficient.

Main characteristics:
- impressive performances for its size (beating https://huggingface.co/meta-llama/Llama-2-7b and https://huggingface.co/huggyllama/llama-13b)
- combine Mixture of Attention heads (MoA) and Mixture of MLP Experts (MoE) – 8 experts with 2 being active for each token
- trained on a rather limited 1.25T tokens from publicly available datasets – training recipe follows the MiniCPM's two-phases training method => first time I see this for a 2B+ model
- $100k to train
- open weights - open sharing of recipes - open dataset - open code => ♡
- still interesting room to improve performances (be it only by training longer)

Links:
- report: https://research.myshell.ai/jetmoe
- model: https://huggingface.co/jetmoe/jetmoe-8b
- code: https://github.com/myshell-ai/JetMoE

Note: I actually detailed all of the MiniCPM schedule, Mixture-of-expert (MoE) and many of the datasets used in this work in my recent little guide to building LLMs in 2024, so feel free to check it out if you want to learn more on these topics: https://www.youtube.com/watch?v=2-SPH9hIKT8
🚀
samusenps a9i giux78 dillfrescott clem osanseviero
❤️
samusenps ajibawa-2023 dillfrescott clem osanseviero
https://cdn-avatars.huggingface.co/v1/production/uploads/64aea8ff67511bd3d965697b/Jxn52EmDF5RApJh8antxn.jpeg
Feynman Innovations
ajibawa-2023
user