PreTrain RoBERTa from scratch in Portuguese
RoBERTa for PortugueseCurrently, there is only a very limited amount of RoBERTa or RoBERTa Large model that was trained from scratch for Portuguese on the hub:Hugging Face – The AI community building the future.. For this project, the goal is to create a RoBERTa model for just the Portuguese language.ModelA randomly initialized RoBERTa modelAvailable training scriptsA masked language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.DatasetWe can use dataset from BrWaC (Brazilian Web as Corpus) a large Portuguese corpus or Wikidata in Portuguese.ChallengesData pre-processingSample preparation for LM(Optional) Desired project outcomeThe desired project output is a strong RoBERTa model in Portuguese.
2021-06-27T13:46:48Z
What a cool project! You can count on me.
2021-06-28T16:54:06Z
Sounds great! Finalizing this project as wellOne thing that you should think about is also how to evaluate the pretrained model on a downstream task.paws-x · Datasets at Hugging Faceorxtreme · Datasets at Hugging Facewould make sense I thinkPutting everybody in one group for now. You can check it outhere.
2021-06-28T17:23:04Z
That would be great. Wanna be part of it!!
2021-06-28T20:16:36Z
Wow, how cool this project is!Quite innovative idea.I’ve already signed up to help with the project, you can count on me!Congratulations to those involved!
2021-06-28T20:27:12Z
Thanks for all, at this moment we think to evaluate the pretrained model onHAREMorsquadorLeNER.
2021-06-29T17:58:44Z
Very interesting project! It’ll be great to participate!
2021-07-01T18:58:20Z
diiogo:Thanks for all, at this moment we think to evaluate the pretrained model onHAREMorsquadorLeNER.Good afternoon@diiogo.I tried to put my username in thespreadsheetbut it is now closed.As I’m interested in the subject, I will follow your work, in particular regarding to the evaluation through downstreams tasks of your Portuguese RoBERTa model.Question Answering (QA) with SQuAD 1.1 pt: I published 2 models BERT finetuned with this dataset. That will be interesting to compare your f1 e exact-match (link to the SQuAD 1.1 pt BERT models in the HF model hub).Token Classification with LeNER-br: thedataset is in the datasets hub of HF. I will publish soon a BERT model finetuned with it, too.Text Classification with ASSIN and ASSIN2 (RTE:  Recognizing Textual Entailment): the ASSIN and ASSIN2 (Avaliação de Similaridade Semântica e INferência textual) corpus are corpus annotated with pairs of sentences written in Portuguese that is suitable for the exploration of textual entailment and paraphrasing classifiers. The datasets are in the datasets hub of HF (ASSIN,ASSIN2). I will publish soon a BERT model finetuned with them, too.It will be great to test you model with other datasets and/or tasks. What do you think?
2021-07-02T00:25:36Z
This group has a lot of participationGiving you guys directly to TPUs tomorrow! Split the team randomly into two in the official google sheet, but this shouldn’t change anything - just that you have access two 2 TPU v3-8sMight make organization a bit easier to split work on two VMs!
2021-07-02T12:53:09Z
Good Morning@pierreguillou.Its a nice idea to include your results in evaluate of this task (squad).I think too, we will evaluate the model in other tasks as well.
2021-07-02T12:53:31Z
thank you so much
2021-07-05T13:35:49Z
Hello@diiogo.I edited mypostwith datasets ASSIN and ASSIN2 for Text Classification task.
2021-07-05T14:11:48Z
diiogo:DatasetWe can use dataset from BrWaC (Brazilian Web as Corpus) a large Portuguese corpus or Wikidata in Portuguese.TheBrWaCis in the dataset hub of Hugging Face andOSCAR, too.It might be interesting to mix the 2 together for training.About the embeddings and weights initialization, starting from those of a trained RoBERTa model could  help, tooDownload  the model from the HF Model Hub to get the weights.Train a RoBERTa tokenizer on the training dataset to get the vocabulary.Keep the ambeddings of the common tokens between the downloaded model vocabulary and the new one.For the new tokens, initialize them.
2021-09-27T14:30:36Z
Hi! What’s the current status of this initiative?
2021-09-27T18:51:09Z
Hi, now we are doing the training with the fairseq.
2021-09-28T18:16:23Z
Thanks@diiogo,Any particular reason for this change? (Fairseq instead of HF Trainer)
2021-10-04T17:22:32Z
Hi@Piero,Some features was unstable or missing in Flax/JAXKind regards,Diogo