Pretrain GPT-2 from scratch in Mongolian
GPT2 for MongolianGoal is to create a strong language generation model for MongolianSince initial code and data is pretty much written by@patrickvonplatenand other huggingface members, it should not be so hard to get the first sense.ModelRandomly initialized GPT2 modelDatasetsWe can use OSCAR which is available throughdatasetsDatasetsA causal language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.If there is time left, I’d love to try some private crawling and integrate it datasets.Expected OutcomeUnderstandable Mongolian text generation modelChallengesLack of data → OSCAR Mongolian is just 2.2G. Maybe we need to research ways to acquire more data with this.LinksColab Notebook to train CLMOSCARflax code
2021-07-02T02:57:00Z
I am also so interested in pretraining Mongolian Roberta/DistillBERT on OSCAR and fine-tune it on the translated Mongolian SQuAD2.0.
2021-07-02T03:20:43Z
Great! Please registerhere.
2021-07-02T15:29:49Z
Awesome, officially defining it!