

As you can see, only 2 inputs are required for the model in order to compute a loss: input_ids (which are the
input_ids of the encoded input sequence) and labels (which are the input_ids of the encoded
target sequence). The model will automatically create the decoder_input_ids based on the labels, by
shifting them one position to the right and prepending the config.decoder_start_token_id, which for T5 is
equal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate
English to German: ' before encoding it. This will help in improving the performance, as this task prefix was used
during T5's pre-training.
However, the example above only shows a single training example. In practice, one trains deep learning models in
batches. This entails that we must pad/truncate examples to the same length. For encoder-decoder models, one
typically defines a max_source_length and max_target_length, which determine the maximum length of the
input and output sequences respectively (otherwise they are truncated). These should be carefully set depending on
the task.
In addition, we must make sure that padding token id's of the labels are not taken into account by the loss
function. In PyTorch and Tensorflow, this can be done by replacing them with -100, which is the ignore_index
of the CrossEntropyLoss. In Flax, one can use the decoder_attention_mask to ignore padded tokens from
the loss (see the Flax summarization script for details). We also pass
attention_mask as additional input to the model, which makes sure that padding tokens of the inputs are
ignored. The code example below illustrates all of this.
thon