text
"The Case for Co-Designing Model Architectures with Hardware"
"The Case for Co-Designing Model Architectures with Hardware"
new_line
new_line
text
This is a long overdue paper that we have started discussing back when training BLOOM-176.
This is a long overdue paper that we have started discussing back when training BLOOM-176.
new_line
new_line
text
Basically this paper tells you how to design your model's dimensions for an optimal training throughput.
Basically this paper tells you how to design your model's dimensions for an optimal training throughput.
new_line
new_line
text
Fantastic!
Fantastic!
new_line
new_line
text
Yours truly contributed the SwiGLU section ;)
Yours truly contributed the SwiGLU section ;)
new_line
new_line
link
https://twitter.com/QuentinAnthon15/status/1752393989813375119
https://twitter.com/QuentinAnthon15/status/1752393989813375119
new_line
new_line
link
https://arxiv.org/abs/2401.14489
https://arxiv.org/abs/2401.14489
new_line
"The Case for Co-Designing Model Architectures with Hardware"

This is a long overdue paper that we have started discussing back when training BLOOM-176.

Basically this paper tells you how to design your model's dimensions for an optimal training throughput.

Fantastic!

Yours truly contributed the SwiGLU section ;)

https://twitter.com/QuentinAnthon15/status/1752393989813375119

https://arxiv.org/abs/2401.14489

❤️
clem qanthony radames osanseviero abidlabs samusenps jeffboudier efederici merve muhtasham
🤝
clem osanseviero abidlabs merve muhtasham
https://cdn-avatars.huggingface.co/v1/production/uploads/1605114051380-noauth.jpeg
Jeff Boudier
jeffboudier
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1594311341799-5f07383b19cb630495b812cd.jpeg
Stas Bekman
stas
user