PreTrain RoBERTa from scratch in Thai
RoBERTa/BERT for ThaiCurrently, there is only a very limited amount of BERT-like models for Thai on the hub:Hugging Face – The AI community building the future.. For this project, the goal is to create a RoBERTa/BERT model for just the Thai language.ModelA randomly initialized RoBERTa/BERT modelDatasetsOne can make useOSCARthe dataset is also available through thedatasetslibrary here:oscar · Datasets at Hugging Face.Available training scriptsA masked language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.(Optional) Desired project outcomeThe desired project output is a strong RoBERTa/BERT model in Thai.(Optional) ChallengesThe OSCAR dataset might be too small (it has < 20GB of data for Thai). Also it might be importantto find datasets the BERT-like model can be evaluated on after pretraining in Thai. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification scripthere(Optional) Links to read uponThe most important read would be the following colab:Google Colaboratory
2021-06-30T03:00:43Z
Hi there. I have tried with thisblogfor PyTorch before and would love to do this in Flax/Jax tooI will take a look and update the status back soon.
2021-07-02T15:36:21Z
Let’s define it
2021-07-02T16:55:34Z
Alright, as far as I know once LM is ready, we will  start with checking the downstream task to an existing benchmark like thisone