PreTrain RoBERTa from scratch in Indonesian
RoBERTa in IndonesianThere are currently only afewIndonesian RoBERTa models trained from scratch. We aim to create a strong Indonesian RoBERTa language model, to be used for further fine-tuning.LanguageThe model will be trained in Indonesian (Bahasa Indonesia).ModelA randomly initialized RoBERTa model.DatasetsThere is an IndonesianOSCARdataset, which is also available in Hugging Face’sDatasets.Alternatively, there are publicly available datasets such as:Indo4Bby IndoBenchmarkTraining scriptsWe can make use of theFlax MLM script from Hugging Faceto train the model.(Optional) ChallengesThe deduplicated Indonesian OSCAR dataset is of the size 16GB, which may not be the best option available. The Indo4B dataset, on the other hand, sums up a total of 23GB.(Optional) Desired project outcomeThe desired project output is a strong RoBERTa model in Indonesian. To benchmark the results, we can try to fine-tune the output model to downstream Indonesian LM tasks provided byIndoBenchmark.(Optional) ReadsThe following links can be useful to better understand the project and what has previously been done.Google Colaboratory
2021-06-24T18:28:20Z
we could useCC-100: Monolingual Datasets from Web Crawl Data(36GB)The C4 Multilingual Dataset · Discussion #5265 · allenai/allennlp · GitHub(230 GB)as alternative
2021-06-25T02:45:40Z
Awesome, thanks for the suggestions!
2021-06-26T01:37:35Z
Each team will only be provided one TPU-V3 with 4 TPU, will it be enough to train the largest model from scratch?
2021-06-26T08:17:20Z
Correct me if I’m wrong, but what I have read in the official announcement is that we are going to get TPUv3-8 VM, which has 8 cores in total. Though I’m not entirely sure of its maximum capability to train a very large language model, other proposals have similarly proposed to pre-train language models from scratch as well, only in a different language. Let me know if I’m wrong though.Cheers!
2021-06-27T04:41:03Z
Yes, it is TPU V3-8 with 4 TPUs and 8 cores, equivalent to 4 V100 GPUs.Anyway, regardless of it capability we can still give it a try, at least can gain some learning experience.I am in for this project.
2021-06-28T17:03:42Z
Awesome! let’s officially define this projectPutting everybody in the official sheethere. More people can still join! Leave a comment here or on the sheet if you want to change something.