text
Introducing Fineweb-Edu-Fortified: An enhanced Fineweb-Edu dataset. 📚
Introducing Fineweb-Edu-Fortified: An enhanced Fineweb-Edu dataset. 📚
new_line
new_line
text
This dataset is tailored for NLP tasks and helps streamline model training by offering a more refined, unique dataset. Perfect for startups and researchers looking for high-quality educational content to train, evaluate, or fine-tune AI models. The dataset is based on the Fineweb-Edu subset of the large Fineweb dataset and includes:
This dataset is tailored for NLP tasks and helps streamline model training by offering a more refined, unique dataset. Perfect for startups and researchers looking for high-quality educational content to train, evaluate, or fine-tune AI models. The dataset is based on the Fineweb-Edu subset of the large Fineweb dataset and includes:
new_line
new_line
text
- Exact-match deduplication across all crawls
- Exact-match deduplication across all crawls
new_line
text
- Embeddings for each row using the TaylorAI/bge-micro model
- Embeddings for each row using the TaylorAI/bge-micro model
new_line
text
- Count column indicating duplication frequency
- Count column indicating duplication frequency
new_line
text
- Includes data from 95 Common Crawl crawls (2013-2024)
- Includes data from 95 Common Crawl crawls (2013-2024)
new_line
text
- Rows have been reduced from 1.279B to 0.324B after deduplication
- Rows have been reduced from 1.279B to 0.324B after deduplication
new_line
text
- It is comprised of ~375B tokens (down from 1,320B in Fineweb-Edu)
- It is comprised of ~375B tokens (down from 1,320B in Fineweb-Edu)
new_line
new_line
text
Access the entire Fineweb-Edu-Fortified dataset on Hugging Face → 
Access the entire Fineweb-Edu-Fortified dataset on Hugging Face → 
resource
https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified
dataset
airtrain-ai/fineweb-edu-fortified
https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified
text
new_line
new_line
text
Try a semantic search demo via this Hugging Face Space → 
Try a semantic search demo via this Hugging Face Space → 
resource
https://huggingface.co/spaces/airtrain-ai/fineweb-edu-fortified-search-demo
space
airtrain-ai/fineweb-edu-fortified-search-demo
https://huggingface.co/spaces/airtrain-ai/fineweb-edu-fortified-search-demo
new_line
new_line
text
Many thanks to the amazing 
Many thanks to the amazing 
mention
@josh-sematic
josh-sematic
text
 for his work on this project, the Fineweb/Fineweb-Edu team at Hugging Face for producing the original datasets and for their support during our work on Fineweb-Edu-Fortified, and also thanks to 
 for his work on this project, the Fineweb/Fineweb-Edu team at Hugging Face for producing the original datasets and for their support during our work on Fineweb-Edu-Fortified, and also thanks to 
mention
@underspirit
underspirit
text
 for pointing out the reduction in dataset size that could be achieved via deduplication. 🤗
 for pointing out the reduction in dataset size that could be achieved via deduplication. 🤗
new_line
new_line
Introducing Fineweb-Edu-Fortified: An enhanced Fineweb-Edu dataset. 📚

This dataset is tailored for NLP tasks and helps streamline model training by offering a more refined, unique dataset. Perfect for startups and researchers looking for high-quality educational content to train, evaluate, or fine-tune AI models. The dataset is based on the Fineweb-Edu subset of the large Fineweb dataset and includes:

- Exact-match deduplication across all crawls
- Embeddings for each row using the TaylorAI/bge-micro model
- Count column indicating duplication frequency
- Includes data from 95 Common Crawl crawls (2013-2024)
- Rows have been reduced from 1.279B to 0.324B after deduplication
- It is comprised of ~375B tokens (down from 1,320B in Fineweb-Edu)

Access the entire Fineweb-Edu-Fortified dataset on Hugging Face → https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fortified 

Try a semantic search demo via this Hugging Face Space → https://huggingface.co/spaces/airtrain-ai/fineweb-edu-fortified-search-demo

Many thanks to the amazing @josh-sematic for his work on this project, the Fineweb/Fineweb-Edu team at Hugging Face for producing the original datasets and for their support during our work on Fineweb-Edu-Fortified, and also thanks to @underspirit for pointing out the reduction in dataset size that could be achieved via deduplication. 🤗


🔥
emmanuel-turlay SinclairWang eljanmahammadli LeroyDyer jakemannix debuxe mmhamdy Riswan-Nopiyar osanseviero gabrielmbmb josh-sematic davanstrien VictorSanh nazimali lev-channel djuna
🚀
insumanth mmhamdy josh-sematic davanstrien VictorSanh lev-channel
👍
josh-sematic mmhamdy