How to accessing the input_ids in EvalPrediction.predictions in Seq2SeqTrainer?
When training aSeq2SeqTrainermodel withevaluateand it looks something like:mt_metrics = evaluate.combine(
    ["bleu", "chrf"]
)

def compute_metrics(pred):
    labels_ids = pred.label_ids
    pred_ids = pred.predictions
    
    predictions = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)

    labels_ids[labels_ids == -100] = tokenizer.pad_token_id
    references = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)

    outputs = mt_metrics.compute(predictions=predictions,
                             references=references)

    return outputs

model = MarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-en-de")

training_args = Seq2SeqTrainingArguments(
    output_dir='./',
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    logging_steps=100,
    save_steps=500,
    eval_steps=1,
    max_steps=1_000_000,
    evaluation_strategy="steps",
    predict_with_generate=True,
    report_to=None,
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=valid_data.with_format("torch"),
    eval_dataset=test_data.with_format("torch"),
    compute_metrics=compute_metrics,
)

trainer.train()TheEvalPrediction.predictionsobjects is exposed tocompute_metrics, it contains thelabel_idsand thepredictionsids but it doesn’t contain theinput_ids, sometimes when training computing the metrics that requires theinput_ids:mt_metrics = evaluate.combine(
    ["bleu", "chrf", "comet"]
)

def compute_metrics(pred, input_ids):
    labels_ids = pred.label_ids
    pred_ids = pred.predictions
    
    source = tokenizer.batch_decode(input_ids, skip_special_tokens=True)
    predictions = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)

    labels_ids[labels_ids == -100] = tokenizer.pad_token_id
    references = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)

    outputs = mt_metrics.compute(predictions=predictions,
                             references=references, sources=sources)

    return outputsIs there someway to include theinput_idsin theEvalPredictionobject when usingSeq2SeqTrainer?If there isn’t, could anyone help to point me to docs to rewrite to create my own custom Seq2SeqTrainer and EvalPrediction? Thank you in advance!
2022-11-02T14:02:41Z
You have a training argument for that:include_inputs_for_metrics.
2022-11-02T15:15:55Z
Thanks for the prompt reply! Theinclude_inputs_for_metricsis exactly what’s needed =)
2022-11-20T11:19:01Z
I’m confused, can I know the difference between the three, input, prediction and reference? and what is the evaluation matrics that needs them all?
2022-11-22T01:23:17Z
@inoormoq, here’s a few examples of metrics in theevaluatepackage that uses references. Hope the examples are self explanatoryHuggingface Evaluate for MT Evaluations | Kaggle
2022-11-25T19:27:14Z
That’s very informative in many aspects. Thanks alot.