text
Smol models ftw! AMD released AMD OLMo 1B - beats OpenELM, tiny llama on MT Bench, Alpaca Eval - Apache 2.0 licensed 🔥
Smol models ftw! AMD released AMD OLMo 1B - beats OpenELM, tiny llama on MT Bench, Alpaca Eval - Apache 2.0 licensed 🔥
new_line
new_line
text
> Trained with 1.3 trillion (dolma 1.7) tokens on 16 nodes, each with 4 MI250 GPUs
> Trained with 1.3 trillion (dolma 1.7) tokens on 16 nodes, each with 4 MI250 GPUs
new_line
new_line
text
> Three checkpoints:
> Three checkpoints:
new_line
new_line
text
- AMD OLMo 1B: Pre-trained model
- AMD OLMo 1B: Pre-trained model
new_line
text
- AMD OLMo 1B SFT: Supervised fine-tuned on Tulu V2, OpenHermes-2.5, WebInstructSub, and Code-Feedback datasets
- AMD OLMo 1B SFT: Supervised fine-tuned on Tulu V2, OpenHermes-2.5, WebInstructSub, and Code-Feedback datasets
new_line
text
- AMD OLMo 1B SFT DPO: Aligned with human preferences using Direct Preference Optimization (DPO) on UltraFeedback dataset
- AMD OLMo 1B SFT DPO: Aligned with human preferences using Direct Preference Optimization (DPO) on UltraFeedback dataset
new_line
new_line
text
Key Insights: 
Key Insights: 
new_line
text
> Pre-trained with less than half the tokens of OLMo-1B
> Pre-trained with less than half the tokens of OLMo-1B
new_line
text
> Post-training steps include two-phase SFT and DPO alignment
> Post-training steps include two-phase SFT and DPO alignment
new_line
text
> Data for SFT:
> Data for SFT:
new_line
text
- Phase 1: Tulu V2
- Phase 1: Tulu V2
new_line
text
- Phase 2: OpenHermes-2.5, WebInstructSub, and Code-Feedback
- Phase 2: OpenHermes-2.5, WebInstructSub, and Code-Feedback
new_line
new_line
text
> Model checkpoints on the Hub & Integrated with Transformers ⚡️
> Model checkpoints on the Hub & Integrated with Transformers ⚡️
new_line
new_line
text
Congratulations & kudos to AMD on a brilliant smol model release! 🤗
Congratulations & kudos to AMD on a brilliant smol model release! 🤗
new_line
new_line
resource
https://huggingface.co/collections/amd/amd-olmo-6723e7d04a49116d8ec95070
collection
amd/amd-olmo-6723e7d04a49116d8ec95070
https://huggingface.co/collections/amd/amd-olmo-6723e7d04a49116d8ec95070
Smol models ftw! AMD released AMD OLMo 1B - beats OpenELM, tiny llama on MT Bench, Alpaca Eval - Apache 2.0 licensed 🔥

> Trained with 1.3 trillion (dolma 1.7) tokens on 16 nodes, each with 4 MI250 GPUs

> Three checkpoints:

- AMD OLMo 1B: Pre-trained model
- AMD OLMo 1B SFT: Supervised fine-tuned on Tulu V2, OpenHermes-2.5, WebInstructSub, and Code-Feedback datasets
- AMD OLMo 1B SFT DPO: Aligned with human preferences using Direct Preference Optimization (DPO) on UltraFeedback dataset

Key Insights: 
> Pre-trained with less than half the tokens of OLMo-1B
> Post-training steps include two-phase SFT and DPO alignment
> Data for SFT:
- Phase 1: Tulu V2
- Phase 2: OpenHermes-2.5, WebInstructSub, and Code-Feedback

> Model checkpoints on the Hub & Integrated with Transformers ⚡️

Congratulations & kudos to AMD on a brilliant smol model release! 🤗

https://huggingface.co/collections/amd/amd-olmo-6723e7d04a49116d8ec95070
🚀
AtAndDev John6666 CYGDEN louisbrulenaudet kimleang123 Dolfini Joseph717171 not-lain KvrParaskevi
🔥
AtAndDev CYGDEN Joseph717171 rizky-gumelar not-lain