PreTrain GPT2 from scratch in Russian
GPT2 for RussianCurrently, there is no GPT2 model that was trained from scratch for Turkish on the hub:Hugging Face – The AI community building the future.. For this project, the goal is to create a strong language generation model for Russian.ModelA randomly initialized GPT2 modelDatasetsOne can make useOSCARthe dataset is also available through thedatasetslibrary here:oscar · Datasets at Hugging Face.Available training scriptsA causal language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.(Optional) Desired project outcomeThe desired project output is a GPT2 model that is able to generate Russian language. A nice generation demo can be created for this.(Optional) ChallengesThe dataset on OSCAR is very large: > 300GB. One might want to explore dataset streaming techniques here. Dataset streaming will be merged todatasetsin a couple of days. See PRhereand the docshere.(Optional) Links to read uponThe most important read would be the following colab:Google Colaboratory
2021-07-01T09:48:23Z
I’m interested.