Pretrain RoBERTa-large from scratch in Swedish
Roberta SwedishThe project idea is identical to the one for Pretraining Roberta in Spanish but instead using the Swedish dataset.The idea is to use theSwedish portion of mC4(which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. It might be a good idea to start in 128, as suggestedhere.2. LanguageThe model will be trained in Swedish.3. ModelRoBERTa-large4. DatasetsSwedish portion of mC4 of about 100gb of uncompressed data.5. Training scriptsThere are already Flax scripts to pre-train RoBERTa that we can easily use:https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)6. ChallengesWill the data be enough to be able to train a good model?7. Desired project outcomeA Swedish monolingual well performing model on the usual benchmarks.8. Reads- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf
2021-06-28T17:38:01Z
Just a suggestion: How about we pretrain a common Scandinavian model instead, as our languages are so similar? I’ve made a project suggestion here:Scandinavian RoBERTa.
2021-07-05T14:04:52Z
Officially defining this one as well