Pretrain T5 from scratch in Dutch
There are currently no Dutch seq2seq models on the hub. There exist multilingual seq2seq models such as mbart and mt5, but no Dutch (nl) only models. This project aims to pre-train T5 from scratch using the Dutch segment of mC4.LanguageDutchModelRandomly initialized T5 modelDatasetsOur primary dataset for pre-training is the Dutch part of multilingual C4, and separately downloaded and curated Dutch news site documents from Common Crawl. We re-use these datasets that have been already cleaned for the project in the FlaxJax week that trains BigBird from scratch on Dutch. The cleaning uses an adapted version of the clean script used for the english c4, that filters badwords and content that are not sentences. Details and source of the data cleaning will be provided on the model hub for this project.The dataset for fine-tuning will be CNN and XSUM translated to Dutch, and news summaries from the Dutch nu.nl website downloaded from CommonCrawl.Training scriptThe starting point for the training will betransformers/run_t5_mlm_flax.py at master · huggingface/transformers · GitHubThis will result in a model that needs to be fine-tuned on a downstream task.For the demo we will fine-tune the model to perform summarization of news articles in Dutch usingtransformers/run_summarization_flax.py at 7d6285a921a23c06169e2d90c94faa0d92d00d78 · huggingface/transformers · GitHubChallengesAdapting the training script for our use case and custom datafiles.Desired project outcomeA T5 model that can be finetuned for Dutch seq2seq tasksA T5 model that can summarize news articles in the Dutch languageA streamlit app that demonstrates news article summarization
2021-07-06T18:08:38Z
Same team!
2021-07-07T10:37:06Z
FinallyGreat! Would love to help here