text
Introducing GlotCC: a new 2TB corpus based on an early 2024 CommonCrawl snapshot with data for 1000+ languages.
Introducing GlotCC: a new 2TB corpus based on an early 2024 CommonCrawl snapshot with data for 1000+ languages.
new_line
new_line
text
🤗 corpus v1: 
🤗 corpus v1: 
resource
https://huggingface.co/datasets/cis-lmu/GlotCC-V1
dataset
cis-lmu/GlotCC-V1
https://huggingface.co/datasets/cis-lmu/GlotCC-V1
new_line
text
🐱 pipeline v3: 
🐱 pipeline v3: 
link
https://github.com/cisnlp/GlotCC
https://github.com/cisnlp/GlotCC
new_line
new_line
text
More details? Stay tuned for our upcoming paper.
More details? Stay tuned for our upcoming paper.
new_line
text
More data? In the next version, we plan to include additional snapshots of CommonCrawl.
More data? In the next version, we plan to include additional snapshots of CommonCrawl.
new_line
new_line
text
Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).
Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).
new_line
Introducing GlotCC: a new 2TB corpus based on an early 2024 CommonCrawl snapshot with data for 1000+ languages.

🤗 corpus v1: https://huggingface.co/datasets/cis-lmu/GlotCC-V1
🐱 pipeline v3: https://github.com/cisnlp/GlotCC

More details? Stay tuned for our upcoming paper.
More data? In the next version, we plan to include additional snapshots of CommonCrawl.

Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).

👍
kargaranamir osanseviero eliebak yjernite ayymen