Different classification label produced by 'predictions' and 'label_ids' from Trainer.predict()
What doespredictionsandlabel_idsactually mean fromTrainer.predict()?I trained a multilabel classification model and tested it on a test dataset. TheTrainer.predict.metricsgave me the output below:I thoughtlabel_idsshould be the predicted label so I did a confusion matrix betweenlabel_idsand my testing data. The result shows a perfect prediction with accuracy = 1, recall =1, precision = 1 etc.I realized something was wrong so I computed the label myself with the logit values produced byTrainer.predict.predictions:compute_labels = tf.round(tf.nn.sigmoid(test_prediction.predictions))Running confusion matrix with thecompute_labelsand the test data, I am able to get a reasonable prediction results that replicated the output ofTrainer.predict.metrics(i.e., above image).My question is: What doesTrainer.predict.label_idsmean? Why the output I got from this argument produced a perfect prediction results which was obviously wrong?Thank you in advance
2022-06-28T14:53:22Z
Hello, I too naïvely thought thatTrainer.predict.label_idswere the predicted labels. But they are the actual labels, as we can read inthis tutorial. You need to actually retrieve the predicted labels by yourself usingnp.argmax(test_prediction.predictions, axis=-1). As we can read later on in the tutorial, metrics are computed like so:metric = load_metric("glue", "mrpc")metric.compute(predictions=preds, references=predictions.label_ids)Which confirms thatlabel_idsare the actual labels indeed. I also find this very confusing.