Pretrain RoBERTa-large from scratch in Finnish
Finnish RoBERTA-largeThe project idea is somewhat identical to the one for Pretraining Roberta in Spanish but instead using the Finnish datasetsThe idea is to use theFinnish portion of mC4(which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512.  Also we might try few other datasets (OSCAR, STT, Yle news)2. LanguageThe model will be trained in Finnish.3. ModelRoBERTa-large(Maybe also some other models if time is left)4. DatasetsFinnish portion of mC4 of about 100gbYle news datasetSTT news5. Training scriptsThere are already Flax scripts to pre-train RoBERTa that we can easily use:https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)6. ChallengesWill the data be enough to be able to train a good model?7. Desired project outcomeA Finnish monolingual well performing model on the usual benchmarks. We hope to beat SOTA model for this task. Current SOTA is TurkuNLP/finbert8. Reads- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf
2021-06-29T13:39:55Z
Great! Finalizing this project