Pretrain GPT-J-6B from scratch on Arabic
The idea behind this project is to pretrain a GPT-J-6B on an Arabic corpus using JAX and TPUs of course.Some resources:Automate Text-based Workflows at ScaleGPT-J-6B: 6B JAX-Based Transformer – Aran Komatsuzaki
2021-06-27T21:28:52Z
@yalouiniInteresting how to particpate
2021-06-28T09:43:04Z
I think that your interest has been noted. We will see how projects are created.
2021-06-28T17:25:24Z
Here are more details:Arabic GPT-J-6BTrain a GPT-J-6B on Arabic dataset (Arabic Wikipedia for exmple) using Jax on TPUs.2. LanguageThe model will be trained in Arabic.3. ModelGPT-J-6B4. DatasetsArabic Wikipedia.(TODO: more to come)5. Training scriptsTODO: Update6. ChallengesNo prior experience doing this kind of work, so this will be mostly a discovery.7. Desired project outcomeAn Arabic GPT-J model that is available on HuggingFace.8. ReadsGood blog post about GPT-J-6B
2021-06-28T18:34:45Z
I like this project! Will note it downMaybe we’ll find some more people later on. Note that this project requires you guys to use a non-transformer model definition:GitHub - kingoflolz/mesh-transformer-jax: Model parallel transformers in JAX and Haikuand model-paralellism in JAX (given the size of the model). Nevertheless I think it’s a feasible project!Noting it down
2021-06-28T18:47:17Z
Thanks@patrickvonplatenfor the additional details and links!
2021-07-03T11:17:01Z
Some additional links:Implementation that will be used:GitHub - kingoflolz/mesh-transformer-jax: Model parallel transformers in JAX and HaikuWhere the code will live:GitHub - yassineAlouini/arabic-gpt-j-6b: Fine-tuning the GPT-J-6B model on some Arabic datasets using Jax and its ecosystem.
2021-08-13T20:08:12Z
Why the work on this stopped ?I have seen an repo in Hugging Face with the name arabic-GPT-J-6b and it was empty.I hope the idea is still work on progress
2021-08-15T15:23:48Z
Hello@MohamedRashad.For many reasons among which personal reasons but hopefully I will give it more time later this year.Notice that the github repo isn’t empty. For instance this branch:GitHub - yassineAlouini/arabic-gpt-j-6b at fine-tune.Hopefully will have time later but for now it is on hold for sure.