PreTrain T5 for Italian 🇮🇹
Train T5 from scratch in Italian on mC4Currently theItalian Hubhas only autoencoding (e.g.bert-base-italian,BERTino) and autoregressive (e.g.GePpeTto) language models. For this project, the goal is to create a strongText2TextGenerationmodel for seq2seq tasks in Italian.ModelWe’ll be using a randomT5model with a configuration similar to the one oft5-base, since the Flax implementation was recently made available on the Hub.DatasetsRecently themultilingual C4 datasetwas made available, and it surpasses by a large margin the size of the OSCAR corpus (590GB cleaned vs. 69GB). Given the newstreaming capabilities for large datasetsthis should be easily feasible. Another option is to use multiple large datasets (mC4, OSCAR, Italian Wikipedia) withìnterleave_datasetstomix them.Training scriptsWe can leverage the T5 pretraining script that iscurrently being preparedto train the model, andrun_summarization_flax.pyto test downstream performances, e.g. onSQUAD-itconverted in text2text format.ChallengesThe project involves a pre-training of a large LM, so it will inevitably take some time to reach competitive performances. Most scripts and implementations are very new, and as such it is possible that we’ll incur in some issues to be fixed.Desired project outcomeThe main project outcome, besides producing a trained T5 model for Italian, is to have fun and get our hands dirty with some :jax:! We could optionally test the validity of our results at a later stage by fine-tuning the model on some downstream tasks.ReadsExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
2021-06-29T14:36:51Z
Think this is a cool project - let’s see if we get some more participants
2021-07-05T18:04:14Z
Any chance to get some compute for this one?Everything should work out of the box with scripts, so I think I can handle it solo if TPUs are available!@patrickvonplaten
2021-07-07T12:36:56Z
Let’s add it!