

According to this forum post, task prefixes matter when
(1) doing multi-task training (2) your task is similar or related to one of the supervised tasks used in T5's
pre-training mixture (see Appendix D of the paper for the task prefixes
used).
If training on TPU, it is recommended to pad all examples of the dataset to the same length or make use of
pad_to_multiple_of to have a small number of predefined bucket sizes to fit all examples in. Dynamically padding
batches to the longest example is not recommended on TPU as it triggers a recompilation for every batch shape that is
encountered during training thus significantly slowing down the training. only padding up to the longest example in a
batch) leads to very slow training on TPU.
Inference
At inference time, it is recommended to use [~generation.GenerationMixin.generate]. This
method takes care of encoding the input and feeding the encoded hidden states via cross-attention layers to the decoder
and auto-regressively generates the decoder output. Check out this blog post to know all the details about generating text with Transformers.
There's also this blog post which explains how
generation works in general in encoder-decoder models.
thon