PreTrain RoBERTa from scratch in Marathi
RoBERTa/BERT for MarathiCurrently, there are no BERT-like models for Marathi on the hub. For this project, the goal is to create a RoBERTa/BERT model for just the Marathi language.ModelA randomly initialized RoBERTa/BERT modelDatasetsGitHubThe C4 Multilingual Dataset · Discussion #5265 · allenai/allennlpThe wait has been long, but we are finally able to release the C4 multilingual dataset! We now have almost 27TB of clean-ish data, in 101 different languages (plus the "undetected" langua...One can make useA colossal, cleaned version of Common Crawl’s web crawl corpus - Marathidataset. Marathi corpus is ~70GB so should be great start for training a new language model.Available training scriptsA masked language modeling script for Flax is available - huggingface/transformers repo: examples/flax/language-modeling/run_mlm_flax.py. It can be used pretty much without any required code changes.(Optional) Desired project outcomeThe desired project output is a strong RoBERTa/BERT model in Marathi.(Optional) Links to read uponThe most important read would be the following colab:Download C4 dataset - allenai/allennlp/discussions/5056
2021-07-06T16:44:12Z
We are pre-training RoBERTa for Hindi… maybe I can help…How are we gonna evaluate this? Any benchmark datasets available for Marathi?
2021-07-06T16:49:50Z
Found some relevent work on Marathi language:GitHubGitHub - goru001/nlp-for-marathi: State of the Art Language models and...State of the Art Language models and Classifier for Marathi, which is spoken predominantly by Marathi people of Maharashtra, India - GitHub - goru001/nlp-for-marathi: State of the Art Language mode...indicnlp.ai4bharat.orgIndicBERT | AI4Bharat IndicNLPIndicBERT
2021-07-06T17:10:54Z
Let’s define it
2021-07-06T17:13:44Z
It would be a good start I guess. I will help
2021-07-07T13:08:26Z
@nipunsadvilkar,mc4is now available in HF datasets API as well… We can directly use it.And we got access to TPU VM as well. I have created this channel#roberta-pretraining-marathion the discord. Let’s talk more there.
2021-07-07T13:11:50Z
hassiahk:#roberta-pretraining-marathiGreat! Would you mind talking over slack channel? I will create#roberta-pretraining-marathion slack then
2021-07-07T13:13:36Z
That’s fine as well :))