Pre-train ALBERT from scratch for Persian/Farsi language
1. Pre-train Albert from Scratch for the Persian LanguageCurrently, there are some open-source language models for the Farsi language. We Crawl some new samples as a new dataset which I want to use for pre-training Albert for the farsi language from scratch.2. LanguagePersian/Farsi3. ModelALBERT xLarge4. DatasetsWikipedia dumpCommon Crawl dumprandom web scraps5. Training scriptsThere are already Flax scripts to pre-train Albert that we can easily use:transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub6. ChallengesUsing Dataset pipeline (ETL approach) for new txt crawled file
2021-07-07T14:43:20Z
Sounds amazing, I’m also interested in participating.
2021-07-07T14:44:26Z
thanks sure, We are waiting for accept
2021-07-07T18:06:05Z
Alright finalizing it!
2021-07-08T15:00:41Z
I’m interested to join
2021-07-10T16:21:40Z
I’d be happy to help you with this project!@patrickvonplaten, do I need to request TPU access, or can I collaborate using the credential by one of the team members since I have access to the hf-flax group?