Example of how to pretrain T5?
Is there any codebase in huggingface that could be used to pretrain T5 model? Looking into the examples dir in the repo there is nothing mentioned about T5. Thanks!
2021-03-04T18:11:01Z
Still need help on this…
2021-03-04T18:26:22Z
Hi@mralexis, there’s a GitHub issue that might help you:How do I pre-train the T5 model in HuggingFace library using my own text corpus? · Issue #5079 · huggingface/transformers · GitHubIn particularT5ForConditionalGenerationis probably what you are looking for doing pretraining:T5 — transformers 4.3.0 documentation
2021-03-04T18:39:16Z
@lewtunThanks for the quick reply! I did check it out but there is only a code block on how to calculate the loss for pretraining but no other implementation details which are also critical. Do you know whether there is code on that?
2021-03-04T18:52:39Z
Unfortunately I do not know where one can find a detailed example of T5 pretraining, so pinging@valhallain case he does
2021-03-15T07:41:11Z
Hey guys, sorry about the super late response.T5 pre-training is not implemented withTransformers, AFAIK it’s only available in the original T5 repo.What we need to implement this withTransformersis the T5 style denoising dataset. It’s in my todo-list to implement this hopefully early next month.
2021-04-02T13:15:14Z
Hey!just checking in on that to see if anyone has any updates.Thank you
2021-05-07T20:30:59Z
I am also interested in this and actually have a semi working version (needs more testing) based on the original T5 repo. I’d be happy to work together on this to bring to the transformers library if it is still on the roadmap. Here is the colab with the current implementation:Google Colaboratory(scroll down/CTRL-F for DataCollatorForSeq2SeqMaskLanguageModelingI can also open a PR to start this process if interested.
2021-06-11T21:41:57Z
Any more developments here? My understanding is that we’d have to pre-train using the standardTrainerclass with a custom Data Collator as described by@ncoop57.@valhallawould you be able to help/comment?
2021-09-21T08:37:00Z
Hi@valhalla, any update on this ?
2021-09-21T09:45:32Z
T5 pre-training is now supported in JAX/FLAX. You can check out the example script here:transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub. It actually includes 2 scripts:t5_tokenizer_model.py, to train a T5 tokenizer (i.e. SentencePiece) from scratch.run_t5_mlm_flax.py, to pre-train T5. It’s suited to run on TPUs (for which you can obtain access for free by applying to Google’sTFRC program).@patrickvonplatenalso demonstrates how to run the script inthis video(starts around 13:35).This script was developed for theJAX/FLAX community event. It would be really cool if someone contributes the PyTorch version of it. It would mean translating the script from FLAX to PyTorch, which is probably straightforward.
2022-01-13T10:15:02Z
Hi, I convert the parameters trained from JAX/FLAX to the pytorch version.model = FlaxT5ForConditionalGeneration.from_pretrained(pretrained_path)pt_model = T5ForConditionalGeneration.from_pretrained(tmp_path, from_flax=True)However, some weights of T5ForConditionalGeneration were not initialized from the Flax model.Here are the details.All Flax model weights were used when initializing T5ForConditionalGeneration.Some weights of T5ForConditionalGeneration were not initialized from the Flax model and are newly initialized: [‘decoder.embed_tokens.weight’, ‘encoder.embed_tokens.weight’, ‘lm_head.weight’]You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.I guess these three weights are shared.So, I add three lines before saving the parameters.pt_model.encoder.embed_tokens.weight.data = model.params[‘shared’][‘embedding’]._valuept_model.decoder.embed_tokens.weight.data = model.params[‘shared’][‘embedding’]._valuept_model.lm_head.weight.data = model.params[‘shared’][‘embedding’]._valuept_model.save_pretrained(tmp_path)Is this RIGHT?
2022-03-28T14:11:48Z
@lewtun@valhalla@nielsr@patrickvonplatenI am planing to pretrain multilingual T5 small and/or medium from scratch, i can across this post and the  hugginface implementation for T5, my question is can i use the same pretraining script from T5 , by replace the T5Config with mT5Config ? WOULD THIS WORK ?Also how should the dataset be arranged for multilingual languages pretraining ? should all the langages be arranged in a sequential order where a sequence of one lang followed by another eg: [French, German, Italian] or should all the languages be randomly shuffled ?for the record i am planning to pretrain mT5 on indian languages on the oscar corpus and some additionally sourced text corpus.
2022-04-17T16:07:45Z
@StephennFernandesHi, did it work for you with mT5?
2022-10-17T15:47:27Z
Hello@valhalla, do you have any updates? Thank you in advance
2023-03-16T16:31:17Z
We’ve releasednanoT5which is a minimal codebase that reproduces T5-model (similar to BART) pre-training in PyTorch (not Flax), using Huggingface.You can take a look!Any suggestions are more than welcome.