Pre-train RoBERTa from Scratch for Georgian Language
Pre-train RoBERTa from Scratch for Georgian LanguageCurrently, there are no open-source language models for the Georgian language. I have not so large dataset which I want to use for pre-training RoBERTa for the Georgian language from scratch.2. LanguageThe model will be trained in Georgian Language3. ModelRoBERTa4. Datasetswikipedia dumpCommon Crawl dumprandom web scraps5. Training scriptsThere are already Flax scripts to pre-train RoBERTa that we can easily use:transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub)
2021-07-07T10:34:25Z
cool defined it!