PreTrain GPT2 from scratch in Bengali
GPT2 for BengaliCurrently, there is no GPT2 model that was trained from scratch for Bengali on the hub. For this project, the goal is to create a strong language generation model for Bengali using GPT2 Model.2. LanguageBengali.3. ModelA randomly initialized GPT2 model.4. DatasetsOne can make use of OSCAR the dataset is also available through the datasets library here:oscar · Datasets at Hugging Face. The total Bengali resource in OSCAR is 11 GB.Another source can be the mC4 dataset which is available inAllenAI. The resource size is 29GB.5. Training scriptsA causal language modeling script for Flax is availablehere. It can be tweaked for training GPT2.6. ChallengesFix a good tokenizer that covers Bengali vocabulary properly and make sure that the LM doesn’t become character-level LM.7. Desired project outcomeThe desired project output is a GPT2 model that is able to generate Bengali language.8. ReadsThe most important read would be the following colab:Google ColaboratoryOther reads that might be interesting include:GPT-2 no habla español: Artificial Intelligence, Anglocentrism, and the non-human side of DH | | Vanderbilt Universityhttps://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171
2021-06-23T15:56:39Z
I am also a Bengali speaker. I am in!
2021-06-23T16:22:12Z
A Bengali text generation model would be totally great. This would be revolutionary for Bengali NLP research community!I’m sooo in!!!(I really hope this topic gets selected)
2021-06-25T15:10:20Z
Text generation is the dream.BTW does it have to be GPT-2. Are there more efficient (compute or sample) models now?
2021-06-25T17:20:03Z
This looks like a great topic! Let’s wait if more people are interested until Monday and make this an official team than on Monday
2021-06-27T12:47:43Z
I’d like to join too.@sbmaruf@khalidsaifullaah
2021-06-28T16:56:50Z
Let’s officially define this projectPutting everybody in the official sheethere. More people can still join! Leave a comment here or on the sheet if you want to change something.
2021-07-02T15:07:52Z
I would like to join this project@valhalla@patrickvonplaten
2021-08-19T02:26:18Z
would like to join this project@valhalla