text
Impressed by the work of 
Impressed by the work of 
mention
@guipenedo
guipenedo
text
mention
@hynky
hynky
text
mention
@loubnabnl
loubnabnl
text
mention
@anton-l
anton-l
text
mention
@craffel
craffel
text
mention
@lvwerra
lvwerra
text
mention
@thomwolf
thomwolf
text
 on FineWeb.
 on FineWeb.
new_line
new_line
text
LLMs are only as good as the data they have been trained on, but the crucial aspect of pretraining data remains obscure. Our approach lifts the veil on building high-quality pretraining datasets by sharing every detail about this process to enable a wider community to build on top of it.
LLMs are only as good as the data they have been trained on, but the crucial aspect of pretraining data remains obscure. Our approach lifts the veil on building high-quality pretraining datasets by sharing every detail about this process to enable a wider community to build on top of it.
new_line
new_line
text
- The FineWeb-Edu dataset, which outperforms all openly accessible web datasets in a number of educational benchmarks. We built it by developing a quality classifier using annotations generated by an LLM.
- The FineWeb-Edu dataset, which outperforms all openly accessible web datasets in a number of educational benchmarks. We built it by developing a quality classifier using annotations generated by an LLM.
new_line
new_line
text
- A new technical report explaining in detail how to create a large and high-quality web-scale dataset for LLM pretraining such as FineWeb
- A new technical report explaining in detail how to create a large and high-quality web-scale dataset for LLM pretraining such as FineWeb
new_line
new_line
text
👉 
👉 
resource
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
space
HuggingFaceFW/blogpost-fineweb-v1
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
new_line
Impressed by the work of @guipenedo @hynky @loubnabnl @anton-l @craffel @lvwerra @thomwolf on FineWeb.

LLMs are only as good as the data they have been trained on, but the crucial aspect of pretraining data remains obscure. Our approach lifts the veil on building high-quality pretraining datasets by sharing every detail about this process to enable a wider community to build on top of it.

- The FineWeb-Edu dataset, which outperforms all openly accessible web datasets in a number of educational benchmarks. We built it by developing a quality classifier using annotations generated by an LLM.

- A new technical report explaining in detail how to create a large and high-quality web-scale dataset for LLM pretraining such as FineWeb

👉 https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1

🔥
osanseviero lunarflu YannisTevissen ZennyKenny
➕
osanseviero lunarflu