PreTrain ELECTRA from scratch in Portuguese
ELECTRA for PortugueseCurrently, there is no ELECTRA or ELECTRA Large model that was trained from scratch for Portuguese on the hub:Hugging Face – The AI community building the future.. For this project, the goal is to create a ELECTRA model for just the Portuguese language.ModelA randomly initialized ELECTRA modelAvailable training scriptsA masked language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.(Optional) Desired project outcomeThe desired project output is a strong ELECTRA model in Portuguese.
2021-07-02T15:38:11Z
ELECTRA actually uses a special pretraining script, which would need to be written during the sprint - if you are very motivated I’m willing to confirm this “single-person” project and give you a TPU VM. Please leave a comment on the official googe sheet in this case:Confirmed teams for Flax/JAX community week - Google Sheets
2021-11-08T13:22:56Z
Hi Patrick, I am also interested in training Electra for Portuguese. I don’t know if this is currently under development or already has been done, if not, I would like to implement this pretraining script and train the model for Portuguese.