|
--- |
|
license: bigscience-bloom-rail-1.0 |
|
--- |
|
Multilingual Generative Pretrained Transformer with 176B parameters with capacity for Finnish. |
|
This model is built upon pretrained [BLOOM](https://huggingface.co/bigscience/bloom) which is then further pretrained with a combined ROOTS + Finnish (without weighting) dataset for 40B tokens. |
|
|
|
|
|
**Datasets** |
|
|
|
We used a combination of multiple Finnish resources. |
|
|
|
* Finnish Internet Parsebank https://turkunlp.org/finnish_nlp.html |
|
mC4 multilingual colossal, cleaned Common Crawl https://huggingface.co/datasets/mc4 |
|
* Common Crawl Finnish https://TODO |
|
* Finnish Wikipedia https://fi.wikipedia.org/wiki |
|
* Lönnrot Projekti Lönnrot http://www.lonnrot.net/ |
|
* ePub National library ”epub” collection |
|
* National library ”lehdet” collection |
|
* Suomi24 The Suomi 24 Corpus 2001-2020 http://urn.fi/urn:nbn:fi:lb-2021101527 |
|
* Reddit r/Suomi submissions and comments https://www.reddit.com/r/Suomi |
|
* STT Finnish News Agency Archive 1992-2018 http://urn.fi/urn:nbn:fi:lb-2019041501 |
|
* Yle Finnish News Archive 2011-2018 http://urn.fi/urn:nbn:fi:lb-2017070501 |
|
* Yle Finnish News Archive 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050401 |
|
* Yle News Archive Easy-to-read Finnish 2011-2018 http://urn.fi/urn:nbn:fi:lb-2019050901 |
|
* Yle News Archive Easy-to-read Finnish 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050701 |
|
* [ROOTS](https://arxiv.org/abs/2303.03915) - original BLOOM training corpus |
|
|
|
|
|
|
|
**Sampling ratios for Finnish** |
|
|
|
|Dataset | Chars | Ratio | Weight | W.Ratio | |
|
|----------|--------|---------|--------|---------| |
|
|Parsebank | 35.0B | 16.9\% | 1.5 | 22.7\%| |
|
|mC4-Fi | 46.3B | 22.4\% | 1.0 | 20.0\%| |
|
|CC-Fi | 79.6B | 38.5\% | 1.0 | 34.4\%| |
|
|Fiwiki | 0.8B | 0.4\% | 3.0 | 1.0\%| |
|
|Lönnrot | 0.8B | 0.4\% | 3.0 | 1.0\%| |
|
|Yle | 1.6B | 0.8\% | 2.0 | 1.4\%| |
|
|STT | 2.2B | 1.1\% | 2.0 | 1.9\%| |
|
|ePub | 13.5B | 6.5\% | 1.0 | 5.8\%| |
|
|Lehdet | 5.8B | 2.8\% | 1.0 | 2.5\%| |
|
|Suomi24 | 20.6B | 9.9\% | 1.0 | 8.9\%| |
|
|Reddit-Fi | 0.7B | 0.4\% | 1.0 | 0.3\%| |
|
|**TOTAL** | **207.0B** | **100.0\%** | **N/A** | **100.0\%** | |
|
|
|
And for whole continued pretraining, ROOTS is mixed in. |
|
|
