Using Trainer class with T5 - what is returned in EvalPrediction dict?
Hi,I am trying to finetune T5 using the Trainer class. I understand that Trainer doesn’t work out-of-the-box for seq2seq tasks and saw@patrickvonplaten’shttps://github.com/huggingface/transformers/pull/5840which extends the Trainer class to work for Bert2Bert. Is my understanding correct that the Trainer class appropriately handles training for seq2seq models since the loss is calculated by the model itself, and that the only problem is when returning EvalPredictions for calculating and logging custom validation metrics?If so then I would really appreciate if someone can help me to understand what’s being returned in the EvalPrediction dict for T5, it seems like EvalPrediction.predictions is of size batch_size * max_output_len * model_size (31218), is this the generated prediction in embedding form? If so what is the best way to convert this to prediction ids? I tried naively calling model.lm_head() on it but that didn’t seem to be the correct approach.@valhallaperhaps you can weigh in, I also took a look at your notebook finetuning T5 with Pytorch Lightning but would really like to use the HF Trainer class.Thanks for all help.
2020-09-08T11:35:16Z
Hi@melody-juNot sure what you mean by  "T5 doesn’t work out-of-the-box for seq2seq tasks ". T5 is a seq2seq model and it does work for seq2seq tasks.You can useTrainerfor seq2seq tasks as it is. Patrick’s PR extends it so that generative metrics can be calculated (ROUGE, BLUE etc), it should be okay if you calculate them after training the training is finished.To useTrainerfor T5, thedatasetorcollator(if you are using one) should at least returninput_ids,attention_maskandlabels(set pad tokens to -100 inlabels). The rest will be handled byTrainerThis notebookusesTrainerfor fine-tuning T5.Few things to note about  that notebook,I wrote it before v3.0.0, few things have changed after thatDatCollatoris not aclassanymore, so you won’t need to inherit fromDataCollatorwhen creatingT2TDataCollator. Alsocollate_batchshould be renamed to__call__.lm_lablesis now deprecated, uselabelsinstead.No need to manually add</s>anymore, thetokenizernow does that itself.Also you can use theprepare_seq2seq_batchmethod on toknizer which can take the source and target text and returnsinput_ids,attention_maskandlabels.You can also usefinetune.pyscript fromhereto finetune T5 and other seq2seq models. It’s using PL, and there;s WIP version ofSeq2SeqTrainerinthis PR
2020-09-08T11:38:51Z
For more T5 related tipsT5 Finetuning Tips
2020-09-08T11:47:37Z
valhalla:You can useTrainerfor seq2seq tasks as it is. Patrick’s PR extends it so that generative metrics can be calculated (ROUGE, BLUE etc), it should be okay if you calculate them after training the training is finished.Oops sorry, I meant thatTrainerdoes not work out of the box for seq2seq
2020-09-08T11:48:53Z
It works :). It can not calculate generative metrics right now , but that’s being added inSeq2SeqTrainer.
2020-09-08T11:51:35Z
Gotcha! Looking forward to using Seq2SeqTrainer.In the meantime I would like to calculate validation metrics during training but I don’t understand how to manipulate the output given by Trainer in EvalPrediction as the “prediction”, in order to retrieve the id’s of the generated prediction. What is it actually returning here?
2020-09-08T12:00:34Z
To calculate generative metrics during training either clone Patrics branch orSeq2SeqTrainerPR branch.The defaultTrainerreturns the output of the final LM head layer which is why the shape isbatch_size * max_output_len * vocab_size. The above branches instead calls thegeneratemethod inside the eval loop and returns the generated ids asEvalPrediction.predictionsand the actuallabelsasEvalPrediction.label_ids, which then you can pass to the tokenizersdecodeorbatch_decodemethod in thecompute_metricsfunction  to get the text and calculate metrics like ROUGE, BLUE etc
2021-06-28T17:08:10Z
@melody-judid you manage to compute metrics? I am stuck exactly at the same point as you.model_checkpoint = "distilroberta-base"
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint).to('cuda')

def compute_custom_metric(eval_pred):
    print(eval_pred.predictions.shape) # ==> (3387, 32, 50265) (beach_size * max_output_len * vocal_size)
    print(eval_pred.label_ids.shape) # ==> (3387, 32) (batch_size * max_output_len)
    return {'custom_metric': 0}

trainer = Trainer(
    model = model,
    args = training_args,
    train_dataset = train,
    eval_dataset = validation,
    tokenizer = tokenizer,
    data_collator = data_collator,
    compute_metrics = compute_custom_metric
)

trainer.evaluate()I would like to calculate perplexity, not quite sure how to go about it.@valhallaCan you please share some thought.
2022-02-14T19:44:38Z
@valhallaHow to use this script for t5large?I have a machine with 4x4 P100 GPU’s. I am still fairly new to PL. I think from what all I have tried that t5 large is too large to fit on a single GPU so in order to fine-tune it. I need to implement Shared training to fine T5 large or there is something else I can try?