BERTIN: PreTrain RoBERTa-large from scratch in Spanish
BERTINThe idea is to use theSpanish portion of mC4(which roughly amounts for 1TB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. It might be a good idea to start in 128, as suggestedhere.2. LanguageThe model will be trained in Spanish (regardless of variety).3. ModelRoBERTa-large4. DatasetsSpanish portion of mC4 of about 1TB of uncompressed data.5. Training scriptsThere are already Flax scripts to pre-train RoBERTa that we can easily use:https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)6. ChallengesIt is too much data. We need a way to reduce the amount of data to finish on time. Options:Random sampling.Perplexity sampling using percentiles and a Spanish language model. One option here is to use a 5-gram Kneser-Ney model as implemented in the KenLM library (Heafield, 2011) and released by Facebook.7. Desired project outcomeA Spanish monolingual well performing model on the usual benchmarks.8. Reads- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf
2021-06-23T15:38:58Z
So interesting! I am in!
2021-06-23T15:57:36Z
Me too! Nice idea
2021-06-23T17:36:17Z
Yes! Count me in!
2021-06-24T06:47:50Z
Hey! Awesome project! I’m in
2021-06-24T07:50:27Z
I think you have already a full working team, but just in case you need more people, I’d be willing to help. I don’t know if I could be of much use, but the opportunity to contribute to a model called BERTIN is too good to let it go
2021-06-24T08:31:38Z
Awesome to see so much interest here! We can also make multiple teams for the same project - so feel free to keep posting if you’re interested
2021-06-24T08:32:59Z
For this project we can have at least 2 teams!
2021-06-24T09:37:04Z
This project sounds good! I’d like to help with either this one or theSpanish GPT-2, wherever makes more sense to add one person.
2021-06-24T10:56:29Z
I also think this is a very good effort. If I can be of any help please count me in.
2021-06-24T16:50:11Z
Fantastic. I’d love to participate in one of the teams, if possible. Thanks!
2021-06-25T08:57:20Z
What a good idea. Please, count me in if possible!!
2021-06-27T05:45:07Z
Hello@versae& Team,I am interested to be a part of such an amazing project & team. I will try my best to contribute to the BERTIN model. It would be nice if we could discuss some more learning resources that would be useful for this project.I can work in any time zone that is comfortable for everyone in the team.
2021-06-27T18:56:23Z
I’m overwhelmed by the good reception of the idea. Thanks y’all! If we are awarded the TPUs, we must decide on a communication channel a bit more effective than the forum, maybe. A Slack channel? Discord? E-mail?
2021-06-27T21:40:30Z
It was a really nice idea@versae! If you want we could use the Slack group ofNLP en ESand create a channel for us. Some of the people in this thread are already in the community, if the others wanna join you can use thisinvitation.
2021-06-28T09:27:28Z
This seems like a super interesting project. I would love to be a part of this project.
2021-06-28T15:51:09Z
This sounds awesome! I would be interested in joining as well, if there’s still room for more people!
2021-06-28T16:10:18Z
Super exciting about this project with so much interest! Given that we have so many people, I think it makes sense to officially define 2 groups of 7 people (more people can join later :-))I’ve divided people that are interested into two groups and added all the nameshere.Let me know if you don’t want to be in the team and I take you outI think the two teams should talk to each other/ help each other as much as possible, but as stated on Slack before having a single team of 14 is not efficient IMO. Therefore, I propose that both teams take their own approach to doing the dataset preprocessing, model definition etc…One important aspect that should also  be discussed a bit is how the pretrained RoBERTa model should be evaluated ? Maybe onXTREMEafter pretraining?
2021-06-28T16:25:17Z
XTREME and PAWS-X could be a good starting point
2021-06-28T16:27:51Z
I think XTREME is perfect. Not sure if XTREME superseeds PAWS-X or they are just different.So if I understand this correctly, this means we could pre-train 2 versions of BERTIN?It that’s the case, that’s something the 2 teams need to talk about and agree on. My cents would be to test the 2 different approaches to shrink the corpus size: random sampling, perplexity sampling.
No date available
No reply text available