Josh Bauer

josh-sematic's activity

• All
• Models
• Datasets
• Spaces
• Papers
• Collections
• Community
• Posts
• Upvotes
• Likes

New activity in airtrain-ai/fineweb-edu-fortified 17 days ago

Difference with fineweb-edu-dedup (smol-lm corpus)

2

#106 opened 17 days ago by

ferjorosa

liked a Space about 2 months ago

Running

11

📈🚀

Package Download History

Github Star History, but for PyPI package downloads.

liked a dataset 4 months ago

Josephgflowers/Par-Four-Fineweb-Edu-Fortified

Viewer • Updated 17 days ago • 6.05M • 169 • 6

New activity in airtrain-ai/fineweb-edu-fortified 4 months ago

Can you add more labels to "The composition of fineweb-edu-fortified, produced by automatically clustering a 500k row sample in Airtrain " picture?

1

#105 opened 4 months ago by

Josephgflowers

upvoted a collection 4 months ago

Cool and fun Spaces

Collection

13 items • Updated 1 day ago • 3

reacted to joylarkin's post with 🚀🔥 4 months ago

view post

Post

2630

💬 Chat as a way to query SQL! The Airtrain AI team is happy to share a new Hugging Face Space that lets you interact with Hugging Face Hub datasets using a natural language chatbot. 🤗

Start Exploring 👉 airtrain-ai/hf-dataset-chat-to-sql

This Space is forked from davidberenstein1957/text-to-sql-hub-datasets by  @davidberenstein1957 and features chat capability with improved table naming. The tool works with Hugging Face’s recently released in-browser DuckDB-based SQL query engine for datasets.

🚀

9

9

🔥

5

5

+

Reply

upvoted an article 4 months ago

view article

Article

Introducing the SQL Console on Datasets

Sep 17, 2024

• 20

updated a Space 4 months ago

Running

9

🐥 🔍 🤗

Text To SQL Hub Datasets

liked 3 Spaces 4 months ago

Running

11

🦆

SQL Snippets

Running

9

🐥 🔍 🤗

Text To SQL Hub Datasets

Running

22

🐥 🦙 🤗

Text To SQL Hub Datasets

upvoted a collection 4 months ago

Artificial General Intelligence

Collection

A collection of AGI focused items. • 5 items • Updated Sep 5, 2024 • 1

New activity in airtrain-ai/fineweb-edu-fortified 5 months ago

No "\n\n" in the dataset?!

1

#104 opened 5 months ago by

ymh233

Deduped version of fineweb on HuggingFace yields "This dataset has 218 files that have been marked as unsafe."

1

#103 opened 5 months ago by

egor-pakhomov

reacted to joylarkin's post with 🚀🔥👍 5 months ago

view post

Post

3021

Introducing Fineweb-Edu-Fortified: An enhanced Fineweb-Edu dataset. 📚

This dataset is tailored for NLP tasks and helps streamline model training by offering a more refined, unique dataset. Perfect for startups and researchers looking for high-quality educational content to train, evaluate, or fine-tune AI models. The dataset is based on the Fineweb-Edu subset of the large Fineweb dataset and includes:

- Exact-match deduplication across all crawls
- Embeddings for each row using the TaylorAI/bge-micro model
- Count column indicating duplication frequency
- Includes data from 95 Common Crawl crawls (2013-2024)
- Rows have been reduced from 1.279B to 0.324B after deduplication
- It is comprised of ~375B tokens (down from 1,320B in Fineweb-Edu)

Access the entire Fineweb-Edu-Fortified dataset on Hugging Face → airtrain-ai/fineweb-edu-fortified

Try a semantic search demo via this Hugging Face Space → airtrain-ai/fineweb-edu-fortified-search-demo

Many thanks to the amazing @josh-sematic for his work on this project, the Fineweb/Fineweb-Edu team at Hugging Face for producing the original datasets and for their support during our work on Fineweb-Edu-Fortified, and also thanks to  @underspirit for pointing out the reduction in dataset size that could be achieved via deduplication. 🤗

🔥

16

16

🚀

6

6

👍

2

2

+

Reply

liked a Space 5 months ago

Runtime error

6

📚

Fineweb-edu-fortified Semantic Search Demo

updated a Space 5 months ago

Runtime error

6

📚

Fineweb-edu-fortified Semantic Search Demo