text
Hi Huggingfacers! 
Hi Huggingfacers! 
new_line
new_line
text
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint.  Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint.  Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
new_line
new_line
text
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.  
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.  
new_line
new_line
text
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks! 
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks! 
new_line
new_line
text
Paper:  "Adam-mini: Use Fewer Learning Rates To Gain More" 
Paper:  "Adam-mini: Use Fewer Learning Rates To Gain More" 
link
https://arxiv.org/abs/2406.16793
https://arxiv.org/abs/2406.16793
new_line
new_line
text
Code: 
Code: 
link
https://github.com/zyushun/Adam-mini
https://github.com/zyushun/Adam-mini
new_line
new_line
Hi Huggingfacers! 

Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint.  Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.

The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.  

Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks! 

Paper:  "Adam-mini: Use Fewer Learning Rates To Gain More" https://arxiv.org/abs/2406.16793

Code: https://github.com/zyushun/Adam-mini


🔥
yushun0410 osanseviero DmitryRyumin louisbrulenaudet kristaller486 Ramikan-BR not-lain badayvedat wjmcat FAU57 davide221 gogo8232 cnmoro dillfrescott netynet adamelliotfields cloudyu
🚀
yushun0410 sted97 AdemB jakemannix Ramikan-BR Jaward wjmcat secretmoon dillfrescott
👍
yushun0410 osanseviero Rishav2000 Ramikan-BR dashfunnydashdash dillfrescott meeks342
❤️
AdemB Ramikan-BR ttkciar SicariusSicariiStuff dillfrescott
👀
Ramikan-BR dillfrescott
🧠
Ramikan-BR dillfrescott
https://cdn-avatars.huggingface.co/v1/production/uploads/644cb09a22d211df644a0a6c/v0EHypMU4X3Oxxf3cao_O.png
Júlio César
Ramikan-BR
user