Trainer doesn't get to compute_metrics after upgrading to v4.32
Hi,I upgradedtransformersfromv4.9tov4.32and now the trainer doesn’t get into mycompute_metricsfunction. Nothing has changed in the code besides the upgrade.Debugging resulted in finding the place thecompute_metricsis being skipped-trainer.evaluation_loop(), line 3255:# Metrics!
        if self.compute_metrics is not None and all_preds is not None and all_labels is not None:
            if args.include_inputs_for_metrics:
                metrics = self.compute_metrics(
                    EvalPrediction(predictions=all_preds, label_ids=all_labels, inputs=all_inputs)
                )
            else:
                metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
        else:
            metrics = {}The if condition is False becauseall_labels=None. I think its because of theprediction_step()that happens before, which returnslabels=Noneandloss=None.Would appreciate any help with that!Thanks
2023-11-08T15:48:12Z
Hello,I appear to have the same problem. Any updates?
2023-11-10T14:37:46Z
I have the same issueI am tuning a very basic AutoModelForSequenceClassification for a multi-class problem with 10 labels.
2023-11-10T15:06:49Z
@SantiagoCorley@almoggufor me, the problem occurred in this line in the Trainer’s prediction_step:has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)My problem was that I had set label_names in the TrainingArguments to the keys in my label2id dictionary (the actual names of the labels that I am trying to predict.) However, in the documentation, it says the following about label_names:label_names(List[str],optional) — The list of keys in your dictionary of inputs that correspond to the labels.So, my problem was that I misunderstood the label_names argument.Simply not setting label_names in the TrainingArguments, solved the problem for me
2024-07-02T07:02:33Z
Juunge:the problem occurred in this line in the Trainer’s prediction_step:has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)My problem was that I had set label_names in the TrainingArguments to the keys in my label2id dictionary (the actual names of the labels that I am trying to predict.) However, in the documentation, it says the following about label_names:label_names(List[str],optional) — The list of keys in your dictionary of inputs that correspond to the labels.So, my problem was that I misunderstood the label_names argument.Simply not setting label_names in the TrainingArguments, solved the problem for meThanks a lot for your solution. After I disable the option oflable_names,compute_matricscan be called normally.