PreTrain RoBERTa from scratch in Hindi
RoBERTa/BERT for HindiCurrently, there is only a very limited amount of BERT-like models for Hindi on the hub:Hugging Face – The AI community building the future.. For this project, the goal is to create a RoBERTa/BERT model for just the Hindi language.ModelA randomly initialized RoBERTa/BERT modelDatasetsOne can make useOSCARthe dataset is also available through thedatasetslibrary here:oscar · Datasets at Hugging Face.Available training scriptsA masked language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.(Optional) Desired project outcomeThe desired project output is a strong RoBERTa/BERT model in Hindi.(Optional) ChallengesThe OSCAR dataset might be too small (it has < 10GB of data for Hindi). Also it might be importantto find datasets the BERT-like model can be evaluated on after pretraining in Hindi. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification scripthere(Optional) Links to read uponThe most important read would be the following colab:Google Colaboratory
2021-06-23T17:15:06Z
I am planning to work on this project.So, here is my first questionAs the current dataset size is too small for it to effectively train BERT and planning on building my own dataset, by scrapping website that host content in Hindi.The first thing that comes to mind is scrapping news article, which perpetual source of data, we can also tap into some social networks which have more content in Hindi, such as ShareChat. Hindi/Hindustani Literature websitesDoes this approach sounds reasonable, or is too weak because common crawl could have already covered such websites?
2021-06-23T17:23:51Z
Hi@amankhandeliaFor Hindi, the OSCAR corpus has almost 8GB which will be good-enough for a BERT like model.IndicCorpalso has a Hindi corpus.And then there’smC4which has almost 104GB of Hindi data which should be more than enough for this projectAlso, here’s an interesting insight from CamemBERT paper
2021-06-23T17:26:12Z
Some additional resources for hindi,mC4- 105 GBCC-100: 2.5 GB compressed.I don’t think data will be a problem.
2021-06-23T23:55:17Z
In that case, I stand corrected.What are some downstream tasks with corresponding dataset in Hindi on which we can test such model’s efficacy, see if increasing data volume is having an effect on performance of such tasks?
2021-06-24T00:13:39Z
I can recall at least 6 datasets withHindievaluation.See here in the XTREME Benchmark,https://arxiv.org/pdf/2003.11080.pdf
2021-06-24T05:17:21Z
There’sIndicGLUEbenchmark here which has quite a few downstream tasks.
2021-06-25T16:04:26Z
I am interested to work on this project
2021-06-28T11:02:26Z
I am interested in working on this project. Can I take part?
2021-06-28T11:12:44Z
yes, anyone can participate
2021-06-28T15:55:44Z
Great - let’s officially define this project thenPutting everybody in the official sheethere.@sbmaruf- I’ve put you in the group for now as well, feel free to leave a comment in the google sheet to be removed if you don’t want to be in the group. I wasn’t sure
2021-06-29T08:53:47Z
Hi! I will be doing this for Bangla. Probably it’s better I focus more there.
2021-06-29T18:23:46Z
Hi, I am interested in working on this project. Can I be a participate if slots are open ? BTW I am new to JAX.
2021-06-30T05:35:21Z
Hey everyone who is participating in this project, you can join the#roberta-pretraining-hindiproject over on Discord, we can start formulating how to go about it, all the datasets we will use and any downstream task on which we might wish to check the performance of our trained model
2021-06-30T08:09:14Z
Yes! Added you to the team!And don’t worry about JAX, we have an awesomespeaker line-upthat will cover JAX and we and the community will be there to help with any JAX-related questions
2021-06-30T13:44:34Z
Hey. Which Discord? Can you share me an invite?
2021-06-30T14:03:10Z
Yeah sure,Flax-HuggingFace-Community-Week
2021-06-30T14:06:44Z
Thanks man
2021-06-30T14:07:11Z
Hey, I updated link, shared the generic link, this one will directly land you in the group
2021-06-30T14:21:07Z
I am interested. Pls share the link to join. Thanks.
No date available
No reply text available