text
🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.
new_line
new_line
text
Technical report: 
Technical report: 
resource
https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
space
HuggingFaceFW/blogpost-fineweb-v1
https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
text
new_line
text
Dataset: 
Dataset: 
resource
https://hf.co/datasets/HuggingFaceFW/fineweb-edu
dataset
HuggingFaceFW/fineweb-edu
https://hf.co/datasets/HuggingFaceFW/fineweb-edu
text
new_line
new_line
text
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 
new_line
new_line
text
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
new_line
new_line
text
Enjoy!
Enjoy!
🍷 FineWeb technical report is out and so is 📚 FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA.

Technical report: https://hf.co/spaces/HuggingFaceFW/blogpost-fineweb-v1  
Dataset: https://hf.co/datasets/HuggingFaceFW/fineweb-edu 

We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens. 

You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.

Enjoy!
🔥
mmhamdy alielfilali01 GPT007 arjunguha hiauiarau nicolay-r privategeek24 Ariel323 maywell guipenedo neuralink asaduzzaman319
👍
dillfrescott guipenedo neuralink
🧠
neuralink louisbrulenaudet