PreTrain GPT-2 from scratch for German on novel GC4 dataset
PreTrain GPT-2 from scratch for German on novel GC4 datasetTo the best of our knowledge, there is no GPT-J model trained on a german corpus.2. LanguageThe model will be trained in German.3. ModelGPT-J4. DatasetsThe German colossal, cleaned Common Crawl corpus.GC4 Corpus — German NLP Group documentationAnother potential source of data is that which was used by the Digitale Bibliothek/Münchener Digitalisierungszentrum to pretrain their German GPT-2 variant5. Training scriptsGitHubGitHub - kingoflolz/mesh-transformer-jax: Model parallel transformers in JAX...Model parallel transformers in JAX and Haiku. Contribute to kingoflolz/mesh-transformer-jax development by creating an account on GitHub.6. ChallengesRidiculous amount of data renders both training and preprocessing challenging.7. Desired project outcomeA competitive German language model, and a demo based on the HF SvelteKit inference demos8. Team membersChristopher Akiki (myself) and Alina Mailach (@mailach)
2021-06-25T17:39:15Z
This sounds like a more or less finished team already - coolLet’s note it down on MondayI like the idea and think it’d be cool to make this project happen!GPT-J and GPT2 areverysimilar, so it might be easiest to just use the FlaxGPT2 implementation actuallyIf you think you require model parallelism however, the official GPT-J repo could be the way to goIn any way therun_flax_clm.py:transformers/run_clm_flax.py at master · huggingface/transformers · GitHubshould be helpful hereAlso for the massive amount of data, maybe dataset streaming could help?Load a Dataset in Streaming mode — datasets 1.8.0 documentation
2021-06-28T07:37:51Z
Hi guys, I can provide the data that we’ve used for the GPT-2 variant, if needed(It is “only” 16GB, so it’s ok for a kind of baseline, I think)
2021-06-28T10:20:40Z
Hi@stefan-it! That would befantastic! You’re also more than welcome to join our efforts if you don’t already have a project to work on.We could potentially mix both training sets for better diversity. The Common Crawl does sometimes tend to be noisy, and GC4 includes a quality field of sorts, if i’m not mistaken.
2021-06-28T10:22:34Z
Thank you for the feedback Patrick!I think we would rather work with the code you provided.It also seems that the data already is part of the Hub which makes ingestion/pre-processing considerably easier than we initially thought.german-nlp-group/german_common_crawl · Datasets at Hugging Face
2021-06-29T07:36:15Z
Time to officially define this project!Added everybody in the official sheethere, I also added you @stefan-it, not sure if you’ve decided to join the effort but feel free to leave a comment in the Google sheet and I can remove you if necessary.
2021-06-29T10:02:57Z
Thank you Suzana!
2021-07-02T20:04:23Z
Hi@mailach,I’ve just sent out a mail with the link incl. some thoughts about vocab generation