PreTrain GPT2 from scratch in Spanish
GPT2 for SpanishCurrently, there is no GPT2 model that was trained from scratch for Spanish on the hub:Hugging Face – The AI community building the future.. For this project, the goal is to create a strong language generation model for Spanish.ModelA randomly initialized GPT2 modelDatasetsOne can make useOSCARthe dataset is also available through thedatasetslibrary here:oscar · Datasets at Hugging Face.Available training scriptsA causal language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.(Optional) Desired project outcomeThe desired project output is a GPT2 model that is able to generate Spanish language.(Optional) ChallengesThe dataset on OSCAR is quite large → around 150 GB. One might want to try out dataset streaming to solve this problem. Dataset streaming will be merged todatasetsin a couple of days. See PRhereand the docshere.(Optional) Links to read uponThe most important read would be the following colab:Google ColaboratoryOther reads that might be interesting include:GPT-2 no habla español: Artificial Intelligence, Anglocentrism, and the non-human side of DH | | Vanderbilt Universityhttps://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171
2021-06-23T15:41:11Z
I’m definitely in! Check this@mrm8488it’s exactly what we talked about
2021-06-23T15:43:57Z
Yes, it is! I was thinking about using Spanish portion of mC4 instead of OSCAR. What do you think@patrickvonplatenand@mariagrandury??
2021-06-23T19:57:15Z
Awesome, great that you guys are already so interested in thisThe nice thing about OSCAR is that it is supported in thedatasetslibrary and we now also support datasets streaming:Load a Dataset in Streaming mode — datasets 1.8.0 documentationwhich means that you surely won’t run into any disk space memory problems. We sadly don’t support this out of the box for mC4 (it’s not even indatasetsyet), but I’d definitely recommend to try it out! Maybe you can even first train a model on OSCAR and then a second one on mC4
2021-06-23T20:33:10Z
Great,@patrickvonplaten. We will. So we have the script, the dataset and the knowledge. Let’s start the party!
2021-06-24T09:38:02Z
This sounds good! As I said in the project forBERTIN, I’d like to help in either one.
2021-06-27T05:49:39Z
Hello Team,I am interested to be a part of such an amazing project & team. I will try my best to contribute to the Spanish GPT2 model. It would be nice if we could discuss some more learning resources that would be useful for this project.I can work in any time zone that is comfortable for everyone in the team. It will be interesting to work on both OSCAR & mC4 datasets…
2021-06-28T16:54:36Z
Awesome! let’s officially define this projectPutting everybody in the official sheethere. More people can still join! Leave a comment here or on the sheet if you want to change something.
2021-06-28T23:21:25Z
hi@valhalla,I am definitely interested in joining. Could you add me to the spreadhset?Thanks,
2021-06-28T23:24:30Z
I would love to contribute to this project
2021-06-30T08:53:42Z
@jdposa,@shpotes, you are part of the team
2021-07-01T08:15:32Z
Hey, count with me if possible! I’d like to contribute
2021-07-01T09:43:45Z
you’re in