text
📈 One of the biggest changes in Llama 3 was the training dataset, which grew by 7X over Llama 2 (2T to 15T tokens) 🚀
📈 One of the biggest changes in Llama 3 was the training dataset, which grew by 7X over Llama 2 (2T to 15T tokens) 🚀
new_line
new_line
text
While Meta did not open source the dataset, it sparked a thought... what would happen if everyone had access to a big, high-quality dataset? 🤔
While Meta did not open source the dataset, it sparked a thought... what would happen if everyone had access to a big, high-quality dataset? 🤔
new_line
new_line
text
To address that, in April this year, 
To address that, in April this year, 
mention
@huggingface
huggingface
text
  released FineWeb, a 15T token open-source dataset 🌍
  released FineWeb, a 15T token open-source dataset 🌍
new_line
new_line
text
And now they are releasing FineWeb Technical Report and FineWeb Edu 📚
And now they are releasing FineWeb Technical Report and FineWeb Edu 📚
new_line
new_line
text
🏆 15T tokens in FineWeb outperforming other open datasets
🏆 15T tokens in FineWeb outperforming other open datasets
new_line
text
🎓 1.3T highest-quality educational dataset FineWeb-Edu
🎓 1.3T highest-quality educational dataset FineWeb-Edu
new_line
text
📘 5.4T high-quality educational tokens in FineWeb-Edu-2
📘 5.4T high-quality educational tokens in FineWeb-Edu-2
new_line
new_line
text
FineWeb Edu outperforms other datasets on MMLU, ARC, OpenBookQA 📈
FineWeb Edu outperforms other datasets on MMLU, ARC, OpenBookQA 📈
new_line
new_line
text
ODC-By 1.0 license 📜
ODC-By 1.0 license 📜
new_line
new_line
text
Report: 
Report: 
resource
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
space
HuggingFaceFW/blogpost-fineweb-v1
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
📈 One of the biggest changes in Llama 3 was the training dataset, which grew by 7X over Llama 2 (2T to 15T tokens) 🚀

While Meta did not open source the dataset, it sparked a thought... what would happen if everyone had access to a big, high-quality dataset? 🤔

To address that, in April this year, @huggingface  released FineWeb, a 15T token open-source dataset 🌍

And now they are releasing FineWeb Technical Report and FineWeb Edu 📚

🏆 15T tokens in FineWeb outperforming other open datasets
🎓 1.3T highest-quality educational dataset FineWeb-Edu
📘 5.4T high-quality educational tokens in FineWeb-Edu-2

FineWeb Edu outperforms other datasets on MMLU, ARC, OpenBookQA 📈

ODC-By 1.0 license 📜

Report: https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
🚀
some1nostr osanseviero Best-codes