RobIt : PreTrain RoBERTa-base from scratch in Italian
1. RobItThe idea is to use the Italian portion of mC4 (which is about  590 GB) and OSCAR dataset(which is about 137 GB) to pre-train a RoBERTa-base model.2. LanguageThe model will be trained in Italian3. Model usedROBERTa-base4. DatasetsOSCAR-Italian, Italian Portion ofmultilingual C4 dataset5.Training Scripts:There are already Flax scripts to pre-train RoBERTa that we can easily use:github.comhuggingface/transformersmaster/examples/flax/language-modeling🤗Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.Google Colaboratory6. Challenges:It is too much data. We need a way to reduce the amount of data to finish on time so we will use random sampling.Achieving SOTA.7. Desired Possible Outcome:A Italian monolingual well performing model. We could optionally test the validity of our results at a later stage by fine-tuning the model on some downstream tasks.8. Readshttps://arxiv.org/pdf/1907.11692.pdfWe found a model trained onRoBERTabut we plan on increasing the dataset with a different tokenisation.
2021-06-28T19:51:45Z
Count me in !!!
2021-06-28T19:59:08Z
Count me in
2021-06-29T04:35:54Z
Would love to be a part of this, count me in too!
2021-06-29T13:51:01Z
Great! Finalizing this project!