Pretrain GPT2 from scratch in Korean
Korean-GPTThere are already published Korean GPT2 models, but these models are limited in their use in the industry due to license restrictions. Recently, a large number of South Korean public natural language datasets such as KLUE dataset, AI-Hub dataset, and Modu Corpus have been additionally released. My goal is to train a Korean GPT2 model that anyone can use for any purpose, using only large publicly accessible datasets, including recently added datasets.2. LanguageThe model will be trained in Korean3. ModelGPT24. DatasetsKLUE DatasetKorQuAD DatasetModu Corpus(모두의 말뭉치)Korean Text Datasets from AI-hub+ Maybe extra publicly accessible Korean datasets…https://klue-benchmark.comhttps://korquad.github.iohttps://corpus.korean.go.krhttps://aihub.or.kr5. Training scriptsA causal language modeling script for Flax is availablehere6. ChallengesIt will take time and effort to preprocess each of the various public Korean datasets in the most helpful form for pre-training.
2021-07-01T12:54:39Z
I think this will be an amazing challenge!  Wish I can join this project!
2021-07-01T23:55:22Z
Awesome finalizing this project
2021-07-16T16:29:54Z
! importantAmong the above datasets, I will correct the dataset that I incorrectly labeled as Modu to the official name "NIKL Corpus(국립국어원 모두의 말뭉치) ".