

Training
T5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher
forcing. This means that for training, we always need an input sequence and a corresponding target sequence. The input
sequence is fed to the model using input_ids. The target sequence is shifted to the right, i.e., prepended by a
start-sequence token and fed to the decoder using the decoder_input_ids. In teacher-forcing style, the target
sequence is then appended by the EOS token and corresponds to the labels. The PAD token is hereby used as the
start-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.
One can use [T5ForConditionalGeneration] (or the Tensorflow/Flax variant), which includes the
language modeling head on top of the decoder.