text
✨ Today, we're excited to share the full data processing script used in developing our Sailor models. The repo provides an end-to-end data processing pipeline for LLM training. 🚀
✨ Today, we're excited to share the full data processing script used in developing our Sailor models. The repo provides an end-to-end data processing pipeline for LLM training. 🚀
new_line
new_line
text
💻Code: 
💻Code: 
link
https://github.com/sail-sg/sailcraft
https://github.com/sail-sg/sailcraft
new_line
text
🤗Model: 
🤗Model: 
resource
https://huggingface.co/collections/sail/sailor-language-models-65e19a749f978976f1959825
collection
sail/sailor-language-models-65e19a749f978976f1959825
https://huggingface.co/collections/sail/sailor-language-models-65e19a749f978976f1959825
new_line
text
📜Paper: 
📜Paper: 
resource
https://huggingface.co/papers/2404.03608
paper
2404.03608
https://huggingface.co/papers/2404.03608
Sailor: Open Language Models for South-East Asia (2404.03608)
new_line
text
🌐Homepage: 
🌐Homepage: 
link
https://sailorllm.github.io
https://sailorllm.github.io
new_line
new_line
text
# Overview 🔍
# Overview 🔍
new_line
new_line
text
The pipeline consists of 4 stages🧹:
The pipeline consists of 4 stages🧹:
new_line
text
1️⃣ Initial data cleaning
1️⃣ Initial data cleaning
new_line
text
2️⃣ Near deduplication
2️⃣ Near deduplication
new_line
text
3️⃣ Exact deduplication
3️⃣ Exact deduplication
new_line
text
4️⃣ Second round of data cleaning
4️⃣ Second round of data cleaning
new_line
new_line
text
A special focus was given to the data cleaning part of South-East Asian (SEA) languages🌍
A special focus was given to the data cleaning part of South-East Asian (SEA) languages🌍
new_line
new_line
text
# Use Case ✨
# Use Case ✨
new_line
new_line
text
With this codebase, you can clean your own dataset with:
With this codebase, you can clean your own dataset with:
new_line
new_line
text
✅ Get filtered data counts after each processing stage
✅ Get filtered data counts after each processing stage
new_line
text
✅ Easily configure language-specific cleaning rules (we support Arabic, Bengali, Catalan, Spanish, Basque, French, Hindi, Portuguese, Urdu, and optimize for English, Indonesian, Vietnamese, Chinese, Thai, Lao, Malay)
✅ Easily configure language-specific cleaning rules (we support Arabic, Bengali, Catalan, Spanish, Basque, French, Hindi, Portuguese, Urdu, and optimize for English, Indonesian, Vietnamese, Chinese, Thai, Lao, Malay)
new_line
text
✅ Investigate what data was removed at each processing stage
✅ Investigate what data was removed at each processing stage
new_line
new_line
text
# Acknowledgement 🙏
# Acknowledgement 🙏
new_line
new_line
text
The main credit goes to 
The main credit goes to 
mention
@dreamerdeo
dreamerdeo
text
 , the first author of our Sailor paper ❤️! He put in tremendous effort on the data processing pipeline, enabling the model's great performance. We believe the mini repo will be a valuable resource for researchers working on dataset curation for large language models. 🎉
 , the first author of our Sailor paper ❤️! He put in tremendous effort on the data processing pipeline, enabling the model's great performance. We believe the mini repo will be a valuable resource for researchers working on dataset curation for large language models. 🎉
new_line
new_line
text
Sharing the recipe openly aligns with our commitment to open language model development. 💪 And this repo would not have been possible without the contributions from the open community, including the BigScience data cleaning tool, the all-in-one deduplication tool by 
Sharing the recipe openly aligns with our commitment to open language model development. 💪 And this repo would not have been possible without the contributions from the open community, including the BigScience data cleaning tool, the all-in-one deduplication tool by 
mention
@chenghao
chenghao
text
 , and the deduplication project from Google. 🧠
 , and the deduplication project from Google. 🧠
new_line
new_line
text
# What's Next 🚀
# What's Next 🚀
new_line
new_line
text
Share your thoughts or leave any comments on what you'd like the Sailor models to do! We also have some exciting news coming soon, and please stay tuned. 🚄
Share your thoughts or leave any comments on what you'd like the Sailor models to do! We also have some exciting news coming soon, and please stay tuned. 🚄
new_line
✨ Today, we're excited to share the full data processing script used in developing our Sailor models. The repo provides an end-to-end data processing pipeline for LLM training. 🚀

💻Code: https://github.com/sail-sg/sailcraft
🤗Model: https://huggingface.co/collections/sail/sailor-language-models-65e19a749f978976f1959825
📜Paper: https://huggingface.co/papers/2404.03608
🌐Homepage: https://sailorllm.github.io

# Overview 🔍

The pipeline consists of 4 stages🧹:
1️⃣ Initial data cleaning
2️⃣ Near deduplication
3️⃣ Exact deduplication
4️⃣ Second round of data cleaning

A special focus was given to the data cleaning part of South-East Asian (SEA) languages🌍

# Use Case ✨

With this codebase, you can clean your own dataset with:

✅ Get filtered data counts after each processing stage
✅ Easily configure language-specific cleaning rules (we support Arabic, Bengali, Catalan, Spanish, Basque, French, Hindi, Portuguese, Urdu, and optimize for English, Indonesian, Vietnamese, Chinese, Thai, Lao, Malay)
✅ Investigate what data was removed at each processing stage

# Acknowledgement 🙏

The main credit goes to @dreamerdeo , the first author of our Sailor paper ❤️! He put in tremendous effort on the data processing pipeline, enabling the model's great performance. We believe the mini repo will be a valuable resource for researchers working on dataset curation for large language models. 🎉

Sharing the recipe openly aligns with our commitment to open language model development. 💪 And this repo would not have been possible without the contributions from the open community, including the BigScience data cleaning tool, the all-in-one deduplication tool by @chenghao , and the deduplication project from Google. 🧠

# What's Next 🚀

Share your thoughts or leave any comments on what you'd like the Sailor models to do! We also have some exciting news coming soon, and please stay tuned. 🚄

🔥
mmhamdy dreamerdeo ToluClassics chenghao
🚀
mmhamdy dreamerdeo aloobun
👍
szh TheMrguiller Huiyu