PreTrain GPT2 from scratch in Persian
Pre-training GPT2 for Persian:DescriptionWe want to pre-train GPT-2 from scratch, as stated in the OpenAi paper, with high-quality data that will suit text generation task in the Persian language. There is 2 version of GPT-2 which are somehow small model, but we want to take a further step and train GPT-2 on a more extensive data-set and with more diverse and quality text.Data-setAs part of the research effort,  we are preparing a Persian MEgA data-set, including 50~70 GB of text, which is still in progress. We may also use the Oscar data set, which is already available on Huggingface’s data sets.ChallengesBesides creating the data-set, the main challenge is to choose a tokenizer that suits our purpose in the Persian language and doesn’t mess up with the literature. We are still doing various research in this field, but up until now, our pick is the BPE tokenizer which is GPT’s default tokenizer.Training scriptsWe will use some ideas like (here) andhereand also do some modification to suits our purpose.InsightBesides generating high-quality text, with a big GPT model as a reference, we can fine-tune it on several tasks like chatbots(the data for this task will publish soon by our team).
2021-06-28T19:15:55Z
@patrickvonplatenFirst of all, thank you and Huggingface so much for this opportunity.As I mentioned, we already prepare some data, and the rest is still in progress. I was wondering is it possible to add additional data to the OSCAR data-set and what is your insight on doing such a thing.Thanks again
2021-06-28T19:18:51Z
@saiedI’m interested!
2021-06-28T19:25:29Z
@ironcladgeekWellcome,thank you so much for teaming up with me, lad.
2021-06-29T12:50:59Z
Awesome - 2 is definitely enough for nowFinalizing this project!
2021-06-29T13:19:23Z
@patrickvonplatenGreat, Thank you …
2021-06-29T13:21:43Z
Sounds like a good project! In fact, I have some experiences in this case that would help! I’d like to participate.
2021-06-29T18:30:24Z
Wonderfull@m3hrdadfiGreat to have you here
2021-06-30T12:23:42Z
Added you
2021-06-30T13:08:04Z
I’ve created a channel (#gpt-persian) on Discord to talk about this awesome project! Plz, join us!
2021-06-30T16:34:51Z
@mazy1998the link of channel:DiscordDiscord - A New Way to Chat with Friends & CommunitiesDiscord is the easiest way to communicate over voice, video, and text.  Chat, hang out, and stay close with your friends and communities.
2021-07-01T08:41:59Z
DiscordJoin the Flax-HuggingFace-Community-Week Discord Server!Check out the Flax-HuggingFace-Community-Week community on Discord - hang out with 249 other members and enjoy free voice and text chat.@mazy1998can you try this link
2021-07-01T19:04:58Z
Hi guys! I speak Farsi too and would like to contribute in this project, I am a data science lead in Bayer and work on both vision and NLP.
2021-07-01T23:51:23Z
Adding you
2021-07-07T13:07:45Z
Hey Guys,As Persian guy and a Big fan of huggingface pipeline.I’m so interesting to add to this project
2021-07-07T17:33:25Z
If there is an empty spot, I would also like to join!