Confidence Score in Setfit fine-tuned model
Hi Community,I have successfully trained aSetfitModel for multi-label text classification. However, at inference the prediction output doesn’t give confidence score along with the predicted class.Could you please guide me on how I can get confidence score along with the predicted class label.ThanksAbhijit
2022-11-14T07:17:57Z
Hi Abhijit,I am not sure if I am answering your question but if you instantiated a HF trainer class object as saytrainerand your model was calledmodelthen maybe something like this?trainer.model.predict(test_dataset['text']) #gives predicted label with highest prob
trainer.model.predict_proba(test_dataset['text']) #gives all probabilities- for each label
np.argmax(trainer.model.predict_proba(test_dataset['text'])) #gives the label with the highest probThat may not be correct nor answering your question though.
2022-11-14T16:46:30Z
Thanks@AmandaHattawayfor reply. YESSS!!! it does help. “predict_proba” is my guy. I need to write some more code to get my desired output but this will work. Thank you again.
2023-06-21T01:51:21Z
Hi@MathyIIT, I’m still not able to get the predicted probability, could you pls share your piece of code here ?
2023-09-15T06:41:37Z
Hi@MathyIITcan you share your approach, it is the same I am struggling with now
2024-05-31T12:50:04Z
As mentioned, once you get yourmodelout of the trainer, in addition to being able to callmodel.predict()to get the label with the highest confidence interval, you can also callmodel.predict_proba()to get back a list of confidence intervals.