text
🗄️ Massive data release on the HF Hub for 75 languages!
🗄️ Massive data release on the HF Hub for 75 languages!
new_line
new_line
link
https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2
https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2
new_line
new_line
text
In December of last year, HPLT (
In December of last year, HPLT (
link
https://hplt-project.org/
https://hplt-project.org/
text
) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:
) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:
new_line
new_line
code_fence
```python
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
```
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
python
new_line
text
Let's use this big blob of data to build something awesome in our languages! 🥳
Let's use this big blob of data to build something awesome in our languages! 🥳
🗄️ Massive data release on the HF Hub for 75 languages!

https://huggingface.co/datasets/BramVanroy/hplt_monolingual_v1_2

In December of last year, HPLT (https://hplt-project.org/) released version 1.2 of their dataset. It covers web-crawled data of 75 languages!, in the raw format as well as deduplicated and cleaned sections. In total, we're talking about over 40TB of data! This data was already accessible via their website but I figured the accessibility could be improved by an integration with Hugging Face tooling. 🤗 So I added the dataset here to the Hugging Face hub, enabing direct use in your conventional training pipelines for LLMs or other language technologies. The data will automatically be downloaded and optimised with just one line of code:

```python
load_dataset("BramVanroy/hplt_mono_v1_2", "nl_cleaned")
```
Let's use this big blob of data to build something awesome in our languages! 🥳
❤️
stefan-it samusenps ajibawa-2023 clem osanseviero ymoslem damerajee rafaelpierrehf
https://cdn-avatars.huggingface.co/v1/production/uploads/6487239cca30096ea9f52115/HMte9wjKJgfcxsO-5vb_Q.jpeg
dame rajee
damerajee
user