PreTrain Swahili Flax model
Potentially try out a number of model architectures (t5/roberta/gpt2/bigbird…) using datasets such as Oscar, mc5, gdelt,… & test with a fine-tuned task
2021-06-29T14:20:29Z
I really like this idea, especially since models on Swahili are quite sparse! Do you think we could settle on one model? It might make the project much easier
2021-06-29T18:39:08Z
Guess wanted to keep it open if anybody else joined and was keen on a particular Swahili downstream task but yes could settle on a single transformer architecture.
2021-06-30T12:32:31Z
Alright, let me join you in this project - we have way too little model in Swahili so I’m happy trying to help you here
2021-06-30T12:33:02Z
Would be awesome if we manage to find other people to join this project - otherwise It’ll be use two
2021-06-30T12:37:48Z
I think, first we should decide on a model architecture. I would suggest either BERT or GPT2. If we stick to BERT we should also try to find some good downstream data to fine-tune the model on
2021-06-30T12:38:15Z
And it would be great to find some good datasets in Swahili as well
2021-06-30T12:38:48Z
Feel free to also open a discord, we can chat there for more details
2021-06-30T13:52:21Z
will continue to add them here -Flax Swahili Pretraining - Google Sheets
2021-06-30T19:42:50Z
Awesome added you!