Prepare data to fine-tune T5 model on unsupervised objective
Hi, I couldn’t find a way to fine-tune the T5 model on a dataset in a specific domain (let’s say medical domain) using theunsupervisedobjective. Does the current version of Huggingface support this? Basically, all I need is to prepare the dataset to train the T5 model on the unsupervised objective, which could itself be very tricky. Any pointer on this is highly appreciated. P.S: I am looking for something inPyTorchand not Tensorflow.@valhalla@clem
2021-09-27T13:27:14Z
Hi! Have you found a solution? I also can’t figure out how to fine-tune the pretrained model (mT5) on unlabeled domain specific data using Transformer library. Thank you.
2021-11-03T22:53:38Z
Hello@Adi-0-0-Guptaand@naumov-al.There is an overview of the training task of Language Modeling for T5 in theT5 pageon the Hugging Face site atUnsupervised denoising training.And you will get scripts for training as said in this text from Hugging Face:If you’re interested in pre-training T5 on a new corpus, check out therun_t5_mlm_flax.pyscript in the Examples directory.To train the T5 tokenizer vocab on your specific domain, this script should help:t5_tokenizer_model.pySource:Example scriptsin the T5 pagepre-training: therun_t5_mlm_flax.pyscript allows you to further pre-train T5 or pre-train T5 from scratch on your own data. Thet5_tokenizer_model.pyscript allows you to further train a T5 tokenizer or train a T5 Tokenizer from scratch on your own data. Note that Flax (a neural network library on top of JAX) is particularly useful to train on TPU hardware.