text
Still following your human intuition to mix corpora from different sources for pre-training 🧠? Everyone says that data mixture has a big impact on model performance, but how - and why🕵️? Did you know that web corpora are actually highly impactful for downstream tasks 🏆?
Still following your human intuition to mix corpora from different sources for pre-training 🧠? Everyone says that data mixture has a big impact on model performance, but how - and why🕵️? Did you know that web corpora are actually highly impactful for downstream tasks 🏆?
new_line
new_line
text
Check out our preprint "RegMix: Data Mixture as Regression for Language Model Pre-training" 📄
Check out our preprint "RegMix: Data Mixture as Regression for Language Model Pre-training" 📄
new_line
new_line
text
🔬 In this paper, we've proposed an automatic data mixture method RegMix that achieves a 6.3% improvement over human selection on the widely used HellaSwag benchmark - and it only needs a 2% extra training FLOPs! 📈
🔬 In this paper, we've proposed an automatic data mixture method RegMix that achieves a 6.3% improvement over human selection on the widely used HellaSwag benchmark - and it only needs a 2% extra training FLOPs! 📈
new_line
new_line
text
📄 Paper: 
📄 Paper: 
resource
https://huggingface.co/papers/2407.01492
paper
2407.01492
https://huggingface.co/papers/2407.01492
RegMix: Data Mixture as Regression for Language Model Pre-training (2407.01492)
new_line
text
💻 Code: 
💻 Code: 
link
https://github.com/sail-sg/regmix
https://github.com/sail-sg/regmix
new_line
text
📊 Collection: 
📊 Collection: 
resource
https://huggingface.co/collections/sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
collection
sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
https://huggingface.co/collections/sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
new_line
text
 🎮 Demo: 
 🎮 Demo: 
link
https://huggingface.co/spaces/sail/RegMix
https://huggingface.co/spaces/sail/RegMix
new_line
Still following your human intuition to mix corpora from different sources for pre-training 🧠? Everyone says that data mixture has a big impact on model performance, but how - and why🕵️? Did you know that web corpora are actually highly impactful for downstream tasks 🏆?

Check out our preprint "RegMix: Data Mixture as Regression for Language Model Pre-training" 📄

🔬 In this paper, we've proposed an automatic data mixture method RegMix that achieves a 6.3% improvement over human selection on the widely used HellaSwag benchmark - and it only needs a 2% extra training FLOPs! 📈

📄 Paper: https://huggingface.co/papers/2407.01492
💻 Code: https://github.com/sail-sg/regmix
📊 Collection: https://huggingface.co/collections/sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
 🎮 Demo: https://huggingface.co/spaces/sail/RegMix

👍
xszheng2020 osanseviero Taylor658 Ksgk-fy
🔥
xszheng2020
🚀
xszheng2020