PreTrain GPT2 from scratch in Indonesia
GPT2 for IndonesiaCurrently, there is only few GPT2 model that was trained from scratch for Indonesian (small size only) on the hub:Hugging Face – The AI community building the future.. For this project, the goal is to create a strong language generation model for Indonesia.ModelA randomly initialized GPT2 modelDatasetsOne can make useOSCARthe dataset is also available through thedatasetslibrary here:oscar · Datasets at Hugging Face.A portion mc4 [MC4] (The C4 Multilingual Dataset · Discussion #5265 · allenai/allennlp · GitHub) andCC100 [CC-100: Monolingual Datasets from Web Crawl Data] can also be used for this taskAvailable training scriptsA causal language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.(Optional) Desired project outcomeThe desired project output is a GPT2 model that is able to generate Indonesia language.(Optional) ChallengesThe dataset on MC4 and CC100 is quite large → around 350 GB. One might want to try out dataset streaming to solve this problem. Dataset streaming will be merged todatasetsin a couple of days. See PRhereand the docshere.Since mc4 not available in datasets yet, adjustment for streaming dataset could be required(Optional) Links to read uponThe most important read would be the following colab:Google Colaboratoryhttps://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171
2021-06-24T20:11:35Z
Thanks for creating the proposal, I am in.
2021-06-24T23:16:27Z
Thanks for the proposal, count me in!
2021-06-25T07:27:42Z
Thanks for the proposal, I’m in!
2021-06-25T08:01:39Z
Hopefully we can pretrain gpt2-large using the datasets mentioned above within only one weekBtw, to make our discussion easier, we will use again telegram.
2021-06-25T17:30:14Z
Cool idea and great description - we can probably already fix this team on Monday
2021-06-28T23:18:40Z
We received additional 1.2GB dataset of “indonesian-journals”, which is a collection of journals/papers from several indonesian universities.
2021-06-29T07:26:41Z
Awesome, let’s officially define this project!Added everybody in the official sheethere.
2021-06-29T09:18:41Z
Thanks@Suzana. May I ask you to add also@munggokand@ayameRushiato the official sheet please (I can’t update it my self)? Thanks.
2021-06-29T09:30:52Z
May I join too?
2021-06-29T09:35:37Z
Of course
2021-06-29T11:15:41Z
Done!@cahya
2021-06-29T17:58:43Z
Hi, thank you for the proposal, Count me in.
2021-06-30T12:31:09Z
Done!@muhammadfhadli